Analyzing the the NOAA Storm Database

The analysis below is applied on the NOAA Storm database which contains information about natural events occurred in USA between years 1950-2011. After processing and cleaning the data, the results are shown in 3 separate figures which indicate the frequency of events, health problems, and the damage caused to the properties or crops.

Data Processing

First we load the raw data

raw_data <- read.csv(file = "~/R Working Directory/repdata_data_StormData.csv.bz2")

Let’s take a look at first 5 rows

head(raw_data)
##   STATE__           BGN_DATE BGN_TIME TIME_ZONE COUNTY COUNTYNAME STATE  EVTYPE
## 1       1  4/18/1950 0:00:00     0130       CST     97     MOBILE    AL TORNADO
## 2       1  4/18/1950 0:00:00     0145       CST      3    BALDWIN    AL TORNADO
## 3       1  2/20/1951 0:00:00     1600       CST     57    FAYETTE    AL TORNADO
## 4       1   6/8/1951 0:00:00     0900       CST     89    MADISON    AL TORNADO
## 5       1 11/15/1951 0:00:00     1500       CST     43    CULLMAN    AL TORNADO
## 6       1 11/15/1951 0:00:00     2000       CST     77 LAUDERDALE    AL TORNADO
##   BGN_RANGE BGN_AZI BGN_LOCATI END_DATE END_TIME COUNTY_END COUNTYENDN
## 1         0                                               0         NA
## 2         0                                               0         NA
## 3         0                                               0         NA
## 4         0                                               0         NA
## 5         0                                               0         NA
## 6         0                                               0         NA
##   END_RANGE END_AZI END_LOCATI LENGTH WIDTH F MAG FATALITIES INJURIES PROPDMG
## 1         0                      14.0   100 3   0          0       15    25.0
## 2         0                       2.0   150 2   0          0        0     2.5
## 3         0                       0.1   123 2   0          0        2    25.0
## 4         0                       0.0   100 2   0          0        2     2.5
## 5         0                       0.0   150 2   0          0        2     2.5
## 6         0                       1.5   177 2   0          0        6     2.5
##   PROPDMGEXP CROPDMG CROPDMGEXP WFO STATEOFFIC ZONENAMES LATITUDE LONGITUDE
## 1          K       0                                         3040      8812
## 2          K       0                                         3042      8755
## 3          K       0                                         3340      8742
## 4          K       0                                         3458      8626
## 5          K       0                                         3412      8642
## 6          K       0                                         3450      8748
##   LATITUDE_E LONGITUDE_ REMARKS REFNUM
## 1       3051       8806              1
## 2          0          0              2
## 3          0          0              3
## 4          0          0              4
## 5          0          0              5
## 6          0          0              6

If we take a look to variety of event types { table(raw_data$EVTYPE)}, we’d see that there are some mistakes in entering the values(i.e. Spelling mistakes) and also some of them are in uppercase or lowercase. Some have same meaning but have been written differently. So, let’s fix that:

  • Turning every event into lowercase letters
raw_data$EVTYPE <- tolower(raw_data$EVTYPE)
  • Considering the frequency and nature of each event, categorizing them into 26 events:
  1. Flood: an overflow of water onto normally dry land.

  2. Thunderstorm: A storm with thunder and lightning and typically so heavy rain or hail.

  3. Lightning: a visible electrical discharge from a cloud.

  4. Wind: Moving air; can be soft, moderate, or strong.

  5. Extreme Heat

  6. Drought: a deficiency in precipitation over an extended period.

  7. Astronomical Tides: The tidal levels and water motion which would result from Earth’s rotation and gravitational effects of, the Earth, Sun and Moon, without any atmospheric influences.

  8. Avalanche: a mass of snow, ice, and rocks falling rapidly down a mountainside.

  9. Blizzard: a winter storm + wind + reduced visibility.

  10. Fire

  11. Extreme Cold

  12. Fog

  13. Dust

  14. Frost: Different than blizzards and winter storms. Usually occur after snowfalls, winter storms, or rainfalls.

  15. Heavy Rain

  16. Snowfall

  17. Funnel Cloud: a cone-shaped cloud which extends from the base of a cloud towards the ground without actually reaching the surface.

  18. Hail: a form of precipitation consisting of solid ice.

  19. Hurricane: Different than tornado.

  20. Ice Storm: a storm of freezing rain that leaves a coating of ice.

  21. Landslide: a collapse of a mass of earth or rock from a mountain or cliff.

  22. Tornado: a violent destructive whirling wind accompanied by a funnel-shaped cloud.

  23. High Surfs: ocean waves that are larger and more powerful than usual.

  24. Winter Storm: Blizzards are winter storms but more dangerous.

  25. Waterspout: a whirling column of air and water mist

  26. Others

  • Starting from the most frequent ones to least ones:
library(stringr)
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "thunderstorm")] <- "Thunderstorm"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "tornado")] <- "Tornado"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "flood")] <- "Flood"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "wind")] <- "Wind"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "lightning")] <- "Lightning"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "snow")] <- "Snowfall"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "rain")] <- "Heavy Rain"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "winter")] <- "Winter Storm"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "funnel")] <- "Funnel Cloud"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "waterspout")] <- "Waterspout"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "fld")] <- "Flood"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "fire")] <- "Fire"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "blizzard")] <- "Blizzard"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "drought")] <- "Drought"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "ice storm")] <- "Ice Storm"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "heat")] <- "Extreme Heat"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "frost")] <- "Frost"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "fog")] <- "Fog"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "high surf")] <- "High Surf"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "tropical storm")] <- "Hurricane"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "hurricane")] <- "Hurricane"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "cold")] <- "Extreme Cold"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "landslide")] <- "Landslide"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "rip current")] <- "High Surf"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "dust")] <- "Dust"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "avalanche")] <- "Avalanche"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "storm surge")] <- "High Surf"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "tstm")] <- "Thunderstorm"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "hail")] <- "Hail"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "dry")] <- "Drought"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "astronomical")] <- "Astronomical Tides"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "warm")] <- "Extreme Heat"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "wintry")] <- "Winter Storm"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "surf")] <- "High Surf"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "freez")] <- "Frost"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "ice")] <- "Ice Storm"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "tropical depression")] <- "Hurricane"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "sleet")] <- "Hail"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "typhoon")] <- "Hurricane"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "tsunami")] <- "High Surf"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "glaze")] <- "Frost"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "mudslide")] <- "Landslide"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "mud slide")] <- "Landslide"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "icy")] <- "Frost"
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = "hot")] <- "Extreme Heat"
  • Now, let’s turn others into Others category
others <- names(sort(table(raw_data$EVTYPE)))
for (i in others[2:180]) {
  raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = i)] <- "Others"
}
raw_data$EVTYPE[str_detect(string = raw_data$EVTYPE ,pattern = '\\?')] <- "Others"
  • About property and crop damage:

There are k(Thousand), M(Million), B(Billion), and H(Hundred). Let’s make them all uppercase:

raw_data$PROPDMGEXP <- toupper(raw_data$PROPDMGEXP)
raw_data$CROPDMGEXP <- toupper(raw_data$CROPDMGEXP)

We consider other signs as 1-unit. Now, let’s turn these numbers to their real values:

for (i in 1:902297) {
  if (raw_data$PROPDMG[i] == 0)  {
  }
  else if (raw_data$PROPDMGEXP[i] == "K") {
    raw_data$PROPDMG[i] <- raw_data$PROPDMG[i]*1000
  }
  else if (raw_data$PROPDMGEXP[i] == "H") {
    raw_data$PROPDMG[i] <- raw_data$PROPDMG[i]*100
  }
  else if (raw_data$PROPDMGEXP[i] == "M") {
    raw_data$PROPDMG[i] <- raw_data$PROPDMG[i]*1000000
  }
  else if (raw_data$PROPDMGEXP[i] == "B") {
    raw_data$PROPDMG[i] <- raw_data$PROPDMG[i]*1000000000
  }
}

and the same operation for crop damage:

for (k in 1:902297) {
  if (raw_data$CROPDMG[k] == 0) {
  }
  else if (raw_data$CROPDMGEXP[k] == "K") {
    raw_data$CROPDMG[k] <- raw_data$CROPDMG[i]*1000
  }
  else if (raw_data$CROPDMGEXP[k] == "H") {
    raw_data$CROPDMG[k] <- raw_data$CROPDMG[i]*100
  }
  else if (raw_data$CROPDMGEXP[k] == "M") {
    raw_data$CROPDMG[k] <- raw_data$CROPDMG[i]*1000000
  }
  else if (raw_data$CROPDMGEXP[k] == "B") {
    raw_data$CROPDMG[k] <- raw_data$CROPDMG[k]*1000000000
  }
}

Results and Figures

  • Now that we have neat categories, let’s take a look at the frequency of events in a bar plot
library(ggplot2)
g <- ggplot(raw_data, aes(EVTYPE)) + theme(axis.text.x = element_text(angle = 90)) + xlab("Event") + ylab("Count") + ggtitle("The frequency of events in US" , subtitle = "since 1950 to 2011") + scale_y_continuous(labels = scales::comma) + theme(plot.title = element_text(hjust = 0.5)) +theme(plot.subtitle = element_text(hjust = 0.5))
g + geom_bar()

  • Another graph showing each event causing health problems :
library(dplyr)
## 
## Attaching package: 'dplyr'
## The following objects are masked from 'package:stats':
## 
##     filter, lag
## The following objects are masked from 'package:base':
## 
##     intersect, setdiff, setequal, union
Health <- summarise(group_by(.data = raw_data , EVTYPE) , Fatalities = sum(FATALITIES) , Injuries = sum(INJURIES))
f <- ggplot(Health, aes(x = EVTYPE)) + theme(axis.text.x = element_text(angle = 90)) + xlab("Event") + ylab("Count") + ggtitle("Injuries and Fatalities causing by each event" , subtitle = "since 1950 to 2011") + scale_y_continuous(labels = scales::comma) + theme(plot.title = element_text(hjust = 0.5)) +theme(plot.subtitle = element_text(hjust = 0.5))
f + geom_bar(aes(y = Injuries, fill = "Injuries"), stat = "identity", position = "dodge") + geom_bar(aes(y = Fatalities, fill = "Fatalities"), stat = "identity", position = "dodge")

Considering the graph above, we now know that Tornado is the deadliest event.

  • Now for the last part, showing the economic consequences of each event to both properties and crops:
Damage <- summarise(group_by(.data = raw_data , EVTYPE) , PropDMG = sum(PROPDMG) , CropDMG = sum(CROPDMG))
h <- ggplot(Damage, aes(x = EVTYPE)) + theme(axis.text.x = element_text(angle = 90)) + xlab("Event") + ylab("Damage in $") + ggtitle("Economic Consequences causing by each event" , subtitle = "since 1950 to 2011") + scale_y_continuous(labels = scales::comma) + theme(plot.title = element_text(hjust = 0.5)) +theme(plot.subtitle = element_text(hjust = 0.5))
h + geom_bar(aes(y = PropDMG, fill = "PropDMG"), stat = "identity", position = "dodge") + geom_bar(aes(y = CropDMG, fill = "CropDMG"), stat = "identity", position = "dodge")

Considering the graph above, we now know that Flood is the most destructive event.