Effects of Storms on Population Health and Economy in the U.S.

Sypnosis

In this report, we seek to examine which types of events are most harmful to population health and economic consequences from the data, particularly from year 1990 onwards. To investigate this, we obtained the data from the NOAA Storm Database which collects characteristics of major storms and weather events in the U.S. including types of storms, estimates of any fatalities, injuries, and property damage from the year 1950 to end November 2011.

From these data, we found out that hurricane/typhoon and tsunami events have the highest average number of casualties and economical damage per occurrence.

Loading and Processing the Raw Data

From the NOAA Storm Database, we obtained the data that monitored the major storms and weather events in the U.S. Additional documentation of the database is available which can be found at:

• National Weather Service Storm Data Documentation

• National Climatic Data Center Storm Events FAQ

The accuracy of this analysis would heavily depend on the preprocessing and cleaning of the data. The cleaner and processed the data is, the more accurate the analysis and results will be. Hence, I will be putting effort into this section, so please expect this section to be lengthy.

Loading the necessary packages

The packages that are used for this analysis will be loaded.

library(ggplot2)
library(dplyr)
library(tidyr)
library(data.table)
library(lubridate)

Reading the data and exploration

After this is done, we will load the dataset using the fread function from the data.table package which is way faster than the conventional method.

storm <- fread("storm.csv")

Read 0.0% of 967216 rows
Read 32.1% of 967216 rows
Read 50.7% of 967216 rows
Read 67.2% of 967216 rows
Read 78.6% of 967216 rows
Read 88.9% of 967216 rows
Read 902297 rows and 37 (of 37) columns from 0.523 GB file in 00:00:08

Next step would be to get a quick sense of the data.

dim(storm)

[1] 902297     37

head(storm)

str(storm)

Classes ‘data.table’ and 'data.frame':  902297 obs. of  37 variables:
 $ STATE__   : num  1 1 1 1 1 1 1 1 1 1 ...
 $ BGN_DATE  : chr  "4/18/1950 0:00:00" "4/18/1950 0:00:00" "2/20/1951 0:00:00" "6/8/1951 0:00:00" ...
 $ BGN_TIME  : chr  "0130" "0145" "1600" "0900" ...
 $ TIME_ZONE : chr  "CST" "CST" "CST" "CST" ...
 $ COUNTY    : num  97 3 57 89 43 77 9 123 125 57 ...
 $ COUNTYNAME: chr  "MOBILE" "BALDWIN" "FAYETTE" "MADISON" ...
 $ STATE     : chr  "AL" "AL" "AL" "AL" ...
 $ EVTYPE    : chr  "TORNADO" "TORNADO" "TORNADO" "TORNADO" ...
 $ BGN_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
 $ BGN_AZI   : chr  "" "" "" "" ...
 $ BGN_LOCATI: chr  "" "" "" "" ...
 $ END_DATE  : chr  "" "" "" "" ...
 $ END_TIME  : chr  "" "" "" "" ...
 $ COUNTY_END: num  0 0 0 0 0 0 0 0 0 0 ...
 $ COUNTYENDN: logi  NA NA NA NA NA NA ...
 $ END_RANGE : num  0 0 0 0 0 0 0 0 0 0 ...
 $ END_AZI   : chr  "" "" "" "" ...
 $ END_LOCATI: chr  "" "" "" "" ...
 $ LENGTH    : num  14 2 0.1 0 0 1.5 1.5 0 3.3 2.3 ...
 $ WIDTH     : num  100 150 123 100 150 177 33 33 100 100 ...
 $ F         : chr  "3" "2" "2" "2" ...
 $ MAG       : num  0 0 0 0 0 0 0 0 0 0 ...
 $ FATALITIES: num  0 0 0 0 0 0 0 0 1 0 ...
 $ INJURIES  : num  15 0 2 2 2 6 1 0 14 0 ...
 $ PROPDMG   : num  25 2.5 25 2.5 2.5 2.5 2.5 2.5 25 25 ...
 $ PROPDMGEXP: chr  "K" "K" "K" "K" ...
 $ CROPDMG   : num  0 0 0 0 0 0 0 0 0 0 ...
 $ CROPDMGEXP: chr  "" "" "" "" ...
 $ WFO       : chr  "" "" "" "" ...
 $ STATEOFFIC: chr  "" "" "" "" ...
 $ ZONENAMES : chr  "" "" "" "" ...
 $ LATITUDE  : num  3040 3042 3340 3458 3412 ...
 $ LONGITUDE : num  8812 8755 8742 8626 8642 ...
 $ LATITUDE_E: num  3051 0 0 0 0 ...
 $ LONGITUDE_: num  8806 0 0 0 0 ...
 $ REMARKS   : chr  "" "" "" "" ...
 $ REFNUM    : num  1 2 3 4 5 6 7 8 9 10 ...
 - attr(*, ".internal.selfref")=<externalptr> 

Cleaning and preprocessing of data

In this analysis, I would only examine the effects from the year 2000 onwards (i.e. events on and after 01/01/2000) because the consequences of severe storm events that happened way earlier may not have the same impact as it does many decades later. This is due to infrastructures and properties are now way sturdier than before, people are also more knowledgeable and prepared than before. Hence, it is very likely that the average cost and effect of a tornado of a certain magnitude of strength is lesser now than many decades ago.

It would then, be more appropriate, to only use the data that happened in the last 16 years (i.e. from 2000 to end 2011). Let’s filter the data appropriately in the following code below.

storm$BGN_DATE <- with(storm, as.Date(mdy_hms(BGN_DATE)))
storm <- subset(storm, BGN_DATE >= '2000-01-01')
dim(storm)

[1] 523163     37

It seems like only about 380k of entries are recorded from year 1950 to 1999 (50 years). In comparison, there are 523k of entries from 2010 to 2011 (12 years)! This is helpful for us as it would also remove many old entries that probably wouldn’t contain useful information in this analysis.

Next, we would only need certain columns of this dataset for this analysis, so let’s extract those out. After doing so, let’s also have a look at the event types that can happen.

storm <- storm %>% select(EVTYPE, FATALITIES, INJURIES, PROPDMG, PROPDMGEXP, CROPDMG, CROPDMGEXP)
data.frame(table(storm$EVTYPE))

We noticed that there are many categories that could belong together, yet classfied separately due to:

• Singular v.s. Plural form (e.g. ‘Thunderstorm’ v.s. ‘Thunderstorms’)

• Categorized magnitudes (e.g. ‘TSTM Wind (G40)’ v.s. ‘TSTM Wind (G45)’)

• Different namings (e.g. ‘High Surf’, ‘High Surf Advisories’, ‘High Surf Advisory’, ‘Heavy Surf’)

The next chunk of code, will proceed to clean up all these irregularities that may be present and group up as much as possible using regular expressions. Since there is no quick way to go about doing this, this process may be ‘manual’.

storm$EVTYPE[grepl('DUST', storm$EVTYPE)] <- 'DUST STORM'
storm$EVTYPE[grepl('UNUSUALLY|UNSEASONAL|UNSEASONABLY|ABNORMALLY|EXTREME|EXTREMELY|(EXCESSIVE HEAT)|(VERY WARM|DRY)|RECORD|PROLONG|(WARM WEATHER)', storm$EVTYPE)] <- 'EXTREME COLD/HOT/WET/DRY/EVENT' 
storm$EVTYPE[grepl('^DRY|^DROUGHT', storm$EVTYPE)] <- 'DROUGHT/DRY'
storm$EVTYPE[grepl('SURF|(HIGH SEAS)|(HIGH WATER)|SEICHE', storm$EVTYPE)] <- 'HIGH SURF'
storm$EVTYPE[grepl('ASTRONOMICAL|(HIGH SURF)|CURRENT|ROGUE|(ROUGH SEAS)', storm$EVTYPE)] <- 'LOW/HIGH TIDE/SURF/CURRENT'
storm$EVTYPE[grepl('TSTM WIND|(THUNDERSTORM WIND)', storm$EVTYPE)] <- 'THUNDERSTORM WIND'
storm$EVTYPE[grepl('THUNDERSTORM', storm$EVTYPE)] <- 'THUNDERSTORM WIND'
storm$EVTYPE[grepl('FLOOD|FLD|DROWNING', storm$EVTYPE)] <- 'FLOODING'
storm$EVTYPE[grepl('SNOW', storm$EVTYPE)] <- 'SNOW'
storm$EVTYPE[grepl('^COLD', storm$EVTYPE)] <- 'COLD'
storm$EVTYPE[grepl('FREEZ|FROST', storm$EVTYPE)] <- 'FREEZE/FROST'
storm$EVTYPE[grepl('FOG|SMOKE', storm$EVTYPE)] <- 'FOG/SMOKE'
storm$EVTYPE[grepl('^WINTER|WINTRY', storm$EVTYPE)] <- 'WINTER WEATHER'
storm$EVTYPE[grepl('^WIND|(STRONG WIND)|WND|GUSTY|GRADIENT', storm$EVTYPE)] <- 'WIND'
storm$EVTYPE[grepl('HAIL|(ICE STORM)|SLEET|BLIZZARD|GLAZE', storm$EVTYPE)] <- 'BLIZZARD/HAIL'
storm$EVTYPE[grepl('WILD|(BRUSH FIRE)', storm$EVTYPE)] <- 'WILD/FOREST FIRE'
storm$EVTYPE[grepl('TORNADO|WHIRLDWIND', storm$EVTYPE)] <- 'TORNADO'
storm$EVTYPE[grepl('(BLACK ICE)|(ICE ON ROAD)|(ICY ROAD)|(PATCHY ICE)', storm$EVTYPE)] <- 'ICY ROAD'
storm$EVTYPE[grepl('FUNNEL|WALL', storm$EVTYPE)] <- 'FUNNEL CLOUDS'
storm$EVTYPE[grepl('VOLCAN', storm$EVTYPE)] <- 'VOLCANIC ASH'
storm$EVTYPE[grepl('MUD', storm$EVTYPE)] <- 'MUDSLIDE'
storm$EVTYPE[grepl('HURRICANE', storm$EVTYPE)] <- 'HURRICANE/TYPHOON'
storm$EVTYPE[grepl('LAND|AVALANCHE', storm$EVTYPE)] <- 'LANDSLIDE/AVALANCHE'
storm$EVTYPE[grepl('STORM|SURGE|DEPRESSION|RAIN|PRECIPITATION', storm$EVTYPE)] <- 'HEAVY RAIN/STORMS'

Whew! After all that hardwork, now all that’s left is to do a simple quick filter to keep those events that have more than 5 occurrences since events that happen less than that are negligible. In addition, we will also know look at the column for PROPDMGEXP and CROPDMGEXP.

storm <- storm %>% group_by(EVTYPE) %>% filter(n()>=5) %>% ungroup
table(storm$PROPDMGEXP)

            0      B      K      M 
189113      1     29 328458   5550 

table(storm$CROPDMGEXP)

            B      K      M 
250601      4 271351   1195 

It seems like most of the estimates are in thousands (denoted by K), followed by millions (M) and some cases of billions (B), wow those are bad…

However, there are also many entries where there are no denotations under PROPDMGEXP and CROPDMGEXP. Does this mean that the corresponding entries under PROPDMG and CROPDMG are 0 (i.e. no economic damage). Let’s do a quick check on this.

# Segments out those rows with NULL entries in PROPDMGEXP, then check the corresponding PROPDMG entries if they are all equal to 0
all( (storm %>% filter(storm$PROPDMGEXP == ''))$PROPDMG == 0 )

[1] TRUE

# Do the same check for CROPDMGEXP and CROPDMG
all( (storm %>% filter(storm$CROPDMGEXP == ''))$CROPDMG == 0 )

[1] TRUE

Okay! This is good, now we can convert these in terms of thousands (K) without worrying.

The steps in this following chunk code will be:

1. Convert B and M to numerical values in thousands (i.e. B will become 1,000,000 and M will be 1,000) for both PROPDMGEXP and CROPDMGEXP.

2. Convert K to 1 (already in thousand) and those NULL values to 0.

3. Multiply PROPDMGEXP and PROPDMG together to obtain a standardized numerical value that is in thousands. Do the same for CROPDMGEXP and CROPDMG.

4. Combine these 2 columns into 1 so we can use this column to measure economical damage.

# Step 1 & 2
storm$PROPDMGEXP[storm$PROPDMGEXP == '' | storm$PROPDMGEXP == 0] <- 'N'
storm$PROPDMGEXP <- recode(storm$PROPDMGEXP, B = 1000000, M = 1000, K = 1, N = 0)
storm$CROPDMGEXP[storm$CROPDMGEXP == ''] <- 'N'
storm$CROPDMGEXP <- recode(storm$CROPDMGEXP, B = 1000000, M = 1000, K = 1, N = 0)
# Step 3 & 4
storm <- storm %>% mutate(PROP = PROPDMG*PROPDMGEXP, CROP = CROPDMG*CROPDMGEXP) %>% select(-PROPDMG, -PROPDMGEXP, -CROPDMG, -CROPDMGEXP) %>% mutate(DMG = PROP+CROP) %>% select(-PROP,-CROP)

Now that all of these cleaning and preprocessing of data has been completed, we are finally ready to move on to the analysis.

Results

In this analysis, we will look at the top 6 weather events that have occurred by total amount and average amount of economical damage caused. This will also be done similarly to gauge the effects on the population health.

Firstly, let’s look at the top 6 highest frequency of weather events.

storm %>% group_by(EVTYPE) %>% summarise(FREQ = n()) %>% arrange(desc(FREQ))

Effects of weather events on the economy

We should expect that the total amount of economical damage should some what follow the same order since the higher number of occurrences, the higher the chances of economical damage, thus increasing the total amount of economical damage.

storm %>% group_by(EVTYPE) %>% summarise(COST = sum(DMG)) %>% arrange(desc(COST))

This order of results isn’t the same as the number of occurrences! This proves that there are certain events, which may occur less frequently, and yet can cause a high amount of economical damage when they do happen! Just simply look at the event ‘HURRICANE/TYPHOON’, this isn’t even in the top 10 occurrences by frequency, and yet it is so deadly as it comes in second in economical damage ranking.

To put this into perspective, let’s visualize how often they occur and the total amount of economical damage caused in a bar chart. Let’s look at the top 6 events by economical damage.

storm %>% group_by(EVTYPE) %>% summarize(COST = sum(DMG), FREQ = n()) %>% 
    arrange(desc(COST)) %>% 
    slice(1:6) %>% 
    gather(VARIABLE, VALUE, c(COST,FREQ)) %>% 
    ggplot(aes(x=reorder(EVTYPE,-VALUE), y=VALUE, fill = EVTYPE)) + 
    geom_bar(stat='identity') + 
    facet_grid(VARIABLE ~ ., scales = 'free') + 
    theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
    labs(title = 'Distribution of economic costs and frequency of severe weather events',
         x = 'Weather Event', y = 'Frequency / Economic Cost ($k)') + 
    guides(fill = FALSE)

From the graph, it is apparent flooding and hurricane/typhoon are the deadliest in terms of economic damage whenever they occur. On the other hand, heavy rain/storms and blizzard/hail may occur a lot but they do not tend to cause so much damage whenever they do.

These are the top 6 events that have caused the most economical damage displayed together with the frequencies. Yet, if we truly wish to know how deadly a weather event can be whenever it occurs, it would be more accurate to look at the cost per frequency ratio.

storm %>% group_by(EVTYPE) %>% summarize(COST = sum(DMG), FREQ = n()) %>% mutate(RATIO = COST/FREQ) %>% arrange(desc(RATIO))

Well, it is not surprising to see hurricane/typhoon at the top of the list, yet a surprising entrant is Tsunami which comes in second. It is definitely one of the least occurring events, yet it does pose some serious damage to the economy. In addition, there are also other events that were not seen previously such as drought/dry and wild/forest fire.

Let’s not delve any deeper into this analysis, we shall now move on to explore the weather effects on population health in the same way.

Effects of weather events on population health

As we already know the number of occurrences for each weather event, let’s look at the number of fatalities and injuries for each weather event. In this analysis, we shall not separate the fatalities and injuries columns, let’s merge them together to use as the basis for this investigation.

storm %>% mutate(CASUALTIES = FATALITIES + INJURIES) %>% group_by(EVTYPE) %>% summarize(TOTALCAS = sum(CASUALTIES)) %>% arrange(desc(TOTALCAS))

We are now noticing a different order of weather events in terms of population risks. Not surprisingly tornado is one of them, yet now we noticed extreme weather conditions, lightning and heat as some of the new entrants.

Let’s plot a similar graph as above to see the top 6 distribution of the weather events in terms of its population effects and its frequency.

storm %>% mutate(CASUALTIES = FATALITIES + INJURIES) %>% 
    group_by(EVTYPE) %>% summarize(TOTCAS = sum(CASUALTIES), FREQ = n()) %>% 
    arrange(desc(TOTCAS)) %>% 
    slice(1:6) %>% 
    gather(VARIABLE, VALUE, c(TOTCAS,FREQ)) %>% 
    ggplot(aes(x=reorder(EVTYPE,-VALUE), y=VALUE, fill = EVTYPE)) + 
    geom_bar(stat='identity') + 
    facet_grid(VARIABLE ~ ., scales = 'free') + 
    theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
    labs(title = 'Distribution of casualties and frequency of severe weather events',
         x = 'Weather Event', y = 'Frequency / No. of Casualties') + 
    guides(fill = FALSE)

Similarly, we do not see similar distributions between these 2 graphs. Tornado, lightning, extreme weather conditions/occurrences and heat are truly deadly as they can pose really dangerous risks on the public whenever they happen.

Lastly, let’s look at the number of casualties per weather event occurrence ratio.

storm %>% mutate(CASUALTIES = FATALITIES + INJURIES) %>% group_by(EVTYPE) %>% summarize(TOTALCAS = sum(CASUALTIES), FREQ = n()) %>% mutate(RATIO = TOTALCAS/FREQ) %>% arrange(desc(RATIO))

Once again, we see new entrants that top the charts for ratio of number of casualties per occurrence. Hurricane/typhoon and tsunami are now the 2 weather events that cause the most number of casualties whenever they happen.

In addition, we have also noticed that these 2 weather events are the ones that caused the most economical damage whenever they occur as well. Hence, these 2 events are truly dangerous and deadly, and efforts should be put into researching and finding out ways to mitigate their effects.