The goal of this report is to determine the specific weather events that impact the economy and human health to the highest degree. Specifically, there are two questions this report will seek to address:
Data from the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm events database will be used to evaluate the impact of a range of different weather events. This database covers events dating from 1950 through November 2011, but only data since 1996 will be included in this analysis.
Due to time constraints this analysis will be conducted at the US national level and will ignore avenues of investigation related to seasonal patterns, storm intensity, and the effects of inflation on damage estimates.
library(dplyr) # Data manipulation
library(lubridate) # Dates
library(knitr) # Simple HTML tables
library(ggplot2) # Graphics
library(scales) # Graphics
library(grid) # Graphics
library(gridExtra) # More graphicsData will be extracted from the U.S. National Oceanic and Atmospheric Administration’s (NOAA) storm database. Some documentation regarding the database is located here:
One important feature of the raw data is that beginning in 1996 a more standardized approach was taken when recording storm event types. Prior to this time certain events are either less commonly or less consistently recorded. Going back far enough in time will find that some events are excluded all together. For this reason, this analysis will focus only on data collected on or after January 1, 1996.
The first step is to get a local copy of the NOAA weather data. This file is quite large so we only want to get a copy of this data once and store it locally in a format that is quick and easy to read.
weatherDataFile <- "rawWeatherData.rds"
# Read the raw weather data. Since the dataset is quite large
# we'll store the raw data locally as a perm .rds file. Repeated executions
# of this code will begin with reading the .rds file if it is available. Otherwise
# we'll download from scratch and build it if its not available.
if (!file.exists(weatherDataFile)) {
# External data source and local zip file for this project.
externalZipURL <- "https://d396qusza40orc.cloudfront.net/repdata%2Fdata%2FStormData.csv.bz2"
localZipFile <- "___storm_data___.zip"
# Download and read the zipped csv file.
download.file(externalZipURL, localZipFile)
weatherData <- read.csv(localZipFile, sep = ",", header = TRUE,
na.strings = "?", comment.char = "")
saveRDS(weatherData, weatherDataFile) # Save in current working directory.
}
weatherData <- readRDS(weatherDataFile) # This goes pretty fast once the .rds
# file has been created the first time.
glimpse(weatherData)## Observations: 902,297
## Variables: 37
## $ STATE__ (dbl) 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
## $ BGN_DATE (fctr) 4/18/1950 0:00:00, 4/18/1950 0:00:00, 2/20/1951 0:...
## $ BGN_TIME (fctr) 0130, 0145, 1600, 0900, 1500, 2000, 0100, 0900, 20...
## $ TIME_ZONE (fctr) CST, CST, CST, CST, CST, CST, CST, CST, CST, CST, ...
## $ COUNTY (dbl) 97, 3, 57, 89, 43, 77, 9, 123, 125, 57, 43, 9, 73, ...
## $ COUNTYNAME (fctr) MOBILE, BALDWIN, FAYETTE, MADISON, CULLMAN, LAUDER...
## $ STATE (fctr) AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL...
## $ EVTYPE (fctr) TORNADO, TORNADO, TORNADO, TORNADO, TORNADO, TORNA...
## $ BGN_RANGE (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ BGN_AZI (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ BGN_LOCATI (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ END_DATE (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ END_TIME (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ COUNTY_END (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ COUNTYENDN (lgl) NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA,...
## $ END_RANGE (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ END_AZI (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ END_LOCATI (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ LENGTH (dbl) 14.0, 2.0, 0.1, 0.0, 0.0, 1.5, 1.5, 0.0, 3.3, 2.3, ...
## $ WIDTH (dbl) 100, 150, 123, 100, 150, 177, 33, 33, 100, 100, 400...
## $ F (int) 3, 2, 2, 2, 2, 2, 2, 1, 3, 3, 1, 1, 3, 3, 3, 4, 1, ...
## $ MAG (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ FATALITIES (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 4, 0, ...
## $ INJURIES (dbl) 15, 0, 2, 2, 2, 6, 1, 0, 14, 0, 3, 3, 26, 12, 6, 50...
## $ PROPDMG (dbl) 25.0, 2.5, 25.0, 2.5, 2.5, 2.5, 2.5, 2.5, 25.0, 25....
## $ PROPDMGEXP (fctr) K, K, K, K, K, K, K, K, K, K, M, M, K, K, K, K, K,...
## $ CROPDMG (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ CROPDMGEXP (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ WFO (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ STATEOFFIC (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ ZONENAMES (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ LATITUDE (dbl) 3040, 3042, 3340, 3458, 3412, 3450, 3405, 3255, 333...
## $ LONGITUDE (dbl) 8812, 8755, 8742, 8626, 8642, 8748, 8631, 8558, 874...
## $ LATITUDE_E (dbl) 3051, 0, 0, 0, 0, 0, 0, 0, 3336, 3337, 3402, 3404, ...
## $ LONGITUDE_ (dbl) 8806, 0, 0, 0, 0, 0, 0, 0, 8738, 8737, 8644, 8640, ...
## $ REMARKS (fctr) , , , , , , , , , , , , , , , , , , , , , , , ,
## $ REFNUM (dbl) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, ...
To greatly simplify this analysis only certain raw data elements will be kept:
| Variable | Description |
|---|---|
| STATE | The state in which a weather event was recorded |
| BGN_DATE | The beginning timestamp for the event |
| EVTYPE | The type of event |
| FATALITIES | The number of fatalities associated with the event |
| INJURIES | The number of injuries associated with the event |
| PROPDMG | Significant digits for the estimated property damage associated with the event |
| PROPDMGEXP | The magnitude of the property damage estimate |
| CROPDMG | Significant digits for the estimated crop damage associated with the event |
| CROPDMGEXP | The magnitude of the crop damage estimate |
weatherDataSubset <- weatherData %>% tbl_df %>%
select(STATE, BGN_DATE, EVTYPE, FATALITIES, INJURIES,
PROPDMG, PROPDMGEXP, CROPDMG, CROPDMGEXP) %>%
mutate(EVTYPE = as.character(EVTYPE),
PROPDMGEXP = as.character(PROPDMGEXP),
CROPDMGEXP = as.character((CROPDMGEXP)))
glimpse(weatherDataSubset)## Observations: 902,297
## Variables: 9
## $ STATE (fctr) AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL...
## $ BGN_DATE (fctr) 4/18/1950 0:00:00, 4/18/1950 0:00:00, 2/20/1951 0:...
## $ EVTYPE (chr) "TORNADO", "TORNADO", "TORNADO", "TORNADO", "TORNAD...
## $ FATALITIES (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 4, 0, ...
## $ INJURIES (dbl) 15, 0, 2, 2, 2, 6, 1, 0, 14, 0, 3, 3, 26, 12, 6, 50...
## $ PROPDMG (dbl) 25.0, 2.5, 25.0, 2.5, 2.5, 2.5, 2.5, 2.5, 25.0, 25....
## $ PROPDMGEXP (chr) "K", "K", "K", "K", "K", "K", "K", "K", "K", "K", "...
## $ CROPDMG (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
## $ CROPDMGEXP (chr) "", "", "", "", "", "", "", "", "", "", "", "", "",...
From here several new variables will be constructed to further simplify the analysis.
Property and crop damage estimates are stored in a pair of variables. The first contains the significant digits of the estimate while the second contains the magnitude. These quantities will be combined into one single number in order to aid the analysis. The first step is to recode the magnitudes as integers:
magConvert <- data.frame(sym = c("", "K", "M", "B"),
mag = c(0, 3, 6, 9), stringsAsFactors = FALSE)
print(magConvert)## sym mag
## 1 0
## 2 K 3
## 3 M 6
## 4 B 9
The values of mag in magConvert will be joined to the weather data by matching on the value of the variable sym which is contained in PROPDMGEXP for the property damage estimates and in CROPDMGEXP for the crop damage estimates. Once this is done we can compute full damage estimates using the following formulae:
\[\small{\mathsf{ \text{propDamageFull} = PROPDMG \times 10^{\text{propMag}} }}\] \[\small{\mathsf{ \text{cropDamageFull} = CROPDMG \times 10^{\text{cropMag}} }}\]
Some weather data observations will have values for PROPDMGEXP and CROPDMGEXP that do not match the values of the sym variable in the magConvert dataframe. The values of propDamageFull and cropDamageFull will be set to NA for these. We’ll check the number of such cases and eventually drop these cases.
Once the full property and crop damage estimates have been constructed, they will be added together to form a total damage variable that will be used for this analysis. The formula is:
\[\small{\mathsf{ \text{totalDamage} = \text{propDamageFull} + \text{cropDamageFull} }}\]
Similarly for deaths and injuries we’ll compute a total casualties as the sum of deaths and injuries. No rescaling of the raw data is required. The formula is:
\[\small{\mathsf{ \text{totalCasualties} = \text{FATALITIES} + \text{INJURIES} }}\]
The eventYear that each event ocurred will be extracted from the year value of the BGN_DATE variable in the raw data. This field will be used to keep only those rows with an event year of 1996 or later.
A first pass of the raw weather data now contains a small subset of the original variables.
weatherDataModified <- weatherDataSubset %>%
full_join(magConvert, by = c("PROPDMGEXP" = "sym")) %>% rename(propMag = mag) %>%
full_join(magConvert, by = c("CROPDMGEXP" = "sym")) %>% rename(cropMag = mag) %>%
mutate(eventYear = as.integer(year(mdy_hms(BGN_DATE))),
propDamageFull = PROPDMG * (10 ^ propMag),
cropDamageFull = CROPDMG * (10 ^ cropMag),
totalDamage = propDamageFull + propDamageFull,
totalCasualties = FATALITIES + INJURIES) %>%
filter(eventYear >= 1996) %>%
select(STATE, eventYear, EVTYPE, totalDamage, totalCasualties)
glimpse(weatherDataModified)## Observations: 653,530
## Variables: 5
## $ STATE (fctr) AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, A...
## $ eventYear (int) 1996, 1996, 1996, 1996, 1996, 1996, 1996, 1996...
## $ EVTYPE (chr) "WINTER STORM", "TORNADO", "TSTM WIND", "TSTM ...
## $ totalDamage (dbl) 760000, 200000, 6000, 10000, 4000, 0, 800000, ...
## $ totalCasualties (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
The next step is to clean up the weather event types recorded in the raw data.
The raw data are quite messy in regard to the event types that have been recorded. In order to boil the data down to a sensible list for analysis, the following strategy was used. First, a function was defined that matches each raw EVTYPE value with an event category. The matching was done by trial and error and is somewhat subjective. Further, even though there is a standard list of approved categories spelled out in the documentation, there was not enough time to fine tune the matching process. This situation clearly represents an opportunity where improvements could be made to the analysis.
After the function is defined it is applied to each row of the raw data with the result of the function call stored in a new variable called category. It is this new variable that is used to organize the raw data around specified categories of weather events.
eventCategory <- function(eventType){
eventType <- toupper(eventType)
if (grepl("FLOOD|FLD|STREAM|FLASH|URBAN|RISING", eventType) == TRUE)
return("FLOOD")
else if (grepl("TORNADO|FUNNEL|SPOUT|TORN", eventType) == TRUE)
return("TORNADO")
else if (grepl("TSTM|THUNDERSTORM|WALL|COASTAL", eventType) == TRUE)
return("THUNDERSTORM")
else if (grepl("WIND|WND|GUST|DOWNBURST|MICROB", eventType) == TRUE)
return("WIND")
else if (grepl("HURRICANE|TROPICAL|SURGE|SEICHE|TYPHOON|TSU", eventType) == TRUE)
return("HURRICANE")
else if (grepl("HAIL", eventType) == TRUE)
return("HAIL")
else if (grepl("SURF|MARINE|RIP|TIDE|SWELLS|HIGH|SEAS|WAVE", eventType) == TRUE)
return("SURF/MARINE")
else if (grepl("SNOW|BLIZZARD|MIX|SLEET|FROST|GLAZE|AVALAN|ICE|ICY", eventType) == TRUE)
return("ICE/SNOW")
else if (grepl("RAIN|WET|PRECIP|SHOWER", eventType) == TRUE)
return("RAIN")
else if (grepl("WINTER|FREEZ|COLD|LOW|COOL|HYPO", eventType) == TRUE)
return("COLD WEATHER")
else if (grepl("HEAT|WARM|WARMTH|HOT", eventType) == TRUE)
return("HEAT")
else if (grepl("FIRE|SMOKE", eventType) == TRUE)
return("FIRE")
else if (grepl("LANDSLIDE|MUD|LANDSLUMP|ROCK", eventType) == TRUE)
return("LANDSLIDE")
else if (grepl("DROUGHT|DRY|DRI", eventType) == TRUE)
return("DROUGHT")
else if (grepl("FOG|VOG", eventType) == TRUE)
return("FOG")
else if (grepl("DUST|Dust", eventType) == TRUE)
return("DUSTSTORM")
else if (grepl("VOLCANO|ASH|VOLC", eventType) == TRUE)
return("VOLCANO")
else if (grepl("LIGHT|LIGNT", eventType) == TRUE)
return("LIGHTNING")
else return(tolower(eventType))
}
weatherDataModified$category <- sapply(weatherDataModified$EVTYPE, eventCategory)The weather event categories capture nearly all of the cases present in the raw data. There are a few misc. cases with low frequency or low human/economic impact that were not put in to a category. These will be dropped from our downstream analyses.
weatherDataModified <- weatherDataModified %>%
filter((category %in% c("dam break", "beach erosion", "other", "summary")) == FALSE)The addition of the weather event category variable concludes the heavy data processing steps required to get the raw data in shape for subsequent analysis.
glimpse(weatherDataModified)## Observations: 653,471
## Variables: 6
## $ STATE (fctr) AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, AL, A...
## $ eventYear (int) 1996, 1996, 1996, 1996, 1996, 1996, 1996, 1996...
## $ EVTYPE (chr) "WINTER STORM", "TORNADO", "TSTM WIND", "TSTM ...
## $ totalDamage (dbl) 760000, 200000, 6000, 10000, 4000, 0, 800000, ...
## $ totalCasualties (dbl) 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0...
## $ category (chr) "COLD WEATHER", "TORNADO", "THUNDERSTORM", "TH...
Lets take a look at the frequency with which each weather event category occurs in the data. Included will be total damage and total casualty amounts associated with each.
weatherDataSummary <- weatherDataModified %>%
group_by(category) %>%
summarize(count = n(),
grandTotalDamage = sum(totalDamage, na.rm = TRUE),
grandTotalCasualty = sum(totalCasualties, na.rm = TRUE)) %>%
filter(grandTotalDamage + grandTotalCasualty > 0 & count > 1) %>%
arrange(desc(count))
kable(weatherDataSummary,
col.names = c("Category", "Count", "Grand Total Economic Damage (Dollars)", "Grand Total Casualties"),
caption = "Table 1. Category frequency and impact since 1996",
format.args = list(decimal.mark = ".", big.mark = ","))| Category | Count | Grand Total Economic Damage (Dollars) | Grand Total Casualties |
|---|---|---|---|
| THUNDERSTORM | 223,214 | 15,840,538,560 | 5,568 |
| HAIL | 208,217 | 29,190,434,840 | 730 |
| FLOOD | 79,578 | 319,531,685,340 | 9,857 |
| TORNADO | 32,617 | 49,245,634,020 | 22,183 |
| WIND | 26,298 | 10,890,568,080 | 2,201 |
| ICE/SNOW | 23,191 | 9,743,307,600 | 2,667 |
| COLD WEATHER | 19,195 | 3,169,061,300 | 2,111 |
| LIGHTNING | 13,205 | 1,486,154,160 | 4,792 |
| RAIN | 11,838 | 1,172,482,880 | 330 |
| FIRE | 4,199 | 15,521,099,000 | 1,545 |
| HEAT | 2,591 | 18,487,400 | 9,651 |
| DROUGHT | 2,466 | 2,092,202,000 | 4 |
| SURF/MARINE | 2,332 | 238,065,000 | 1,547 |
| FOG | 1,729 | 40,929,000 | 924 |
| HURRICANE | 1,455 | 274,685,735,120 | 2,065 |
| LANDSLIDE | 620 | 653,256,200 | 98 |
| DUSTSTORM | 557 | 12,275,260 | 428 |
| VOLCANO | 29 | 1,000,000 | 0 |
Eighteen weather event categories appear in the summarized data. Table 1. illustrates that THUNDERSTORM and HAIL related events are by far the most common, and roughly 3 times more frequent than the FLOOD category which is the next most common. But while these top 2 ocurring categories are the most common, they are not the most damaging. That honor goes to the three categories of FLOOD, TORNADO, and HURRICANE. Not a very surprising result!
ggplot(data = weatherDataSummary) +
geom_point(aes(x = grandTotalDamage, y = grandTotalCasualty), size = 3, shape = 16, color = "orange") +
geom_text(aes(x = grandTotalDamage, y = grandTotalCasualty, label = category),size = 2.5, hjust = 0, vjust = 0) +
scale_x_log10(breaks = trans_breaks("log10", function(x) 10 ^ x),
labels = trans_format("log10", math_format(1 %*% 10 ^ .x)),
limits = c(1e6, 2 * 1e12)) +
scale_y_log10(breaks = trans_breaks("log10", function(x) 10 ^ x),
labels = trans_format("log10", math_format(1 %*% 10 ^ .x)),
limits = c(1, 2 * 1e6)) +
xlab("Grand Total Economic Damage (Dollars)") + ylab("Grand Total Casualties") +
ggtitle("Figure 1. Impact of weather events \n on the economy and public health since 1996")In Figure 1. we can see that TORNADO-related weather events accounted for almost 5 times more injuries and deaths than any other category of event. With the exception of Hurricane and Flood categories, Tornados also cause roughly 5 times more damage (in economic terms) than any other weather event. Of notable mention are Heat-related events that appear to be as damaging to the health of the population as Flooding!
While Floods and Hurricanes cause 5 times fewer injuries and deaths than Tornados, they do cause roughly 10 times more damage than anything else!
In Figure 2. we can see the gradual increase over time of the frequency of Floods and perhaps a slight increase in Tornadoes Thankfully Hurricane events don’t seem to be on the rise, but this data set does not include any events from 2012 and forward. For reference, Thunderstorm and Hail events are included as well.
weatherDataByYear <- weatherDataModified %>%
filter(category %in% c("TORNADO", "FLOOD", "HURRICANE", "WINTER", "HAIL", "THUNDERSTORM")) %>%
group_by(eventYear, category) %>%
summarize(count = n())
ggplot(data = weatherDataByYear) +
geom_line(aes(x = eventYear, y = count, group = category, color = category)) +
xlab("Year") + ylab("Frequency") +
ggtitle("Figure 2. Increasing frequency of \n the most damaging event categories")It is notable that while Thunderstorms do less economic damage, they are very harmful to the population and approach the impact that Floods have. Its troubling to see their frequency on the rise (with Hail as well).
At the national level its clear that Floods, Tornados, and Hurricanes are associated with the highest degree of economic damage and human suffering:
Though the number of hurricane events is very small in comparison to other categories, their economic and health impact is enormous. At the same time, we see that the frequency of tornados and floods is on the rise, with flooding the number three most frequent event. State and regional planners should begin working on estimating the cost of more future events vs. the cost of mitigation strategies that can be undertaken today.
In light of global warming/global climate change realities, the fact that Heat-related deaths and injuries are so high should be of great concern to planning agencies as well.
Due to time constraints only a cursory analysis of the data was possible. Consequently there are several avenues for future research that would be interesting to carry out and useful at the state and local planning level.
This analysis was completed at the US national level while the raw data are recorded at the latitude/longitude level. A deeper analysis could reveal new patterns and effects by geography. For example, a government or municipal manager working in the midwest would have less interest in the direct impact of hurricanes and perhaps more interest in the increasing frequency of Thunderstorms and Hail.
Since the data are recorded with a date/time stamp, it would be interesting to understand if there are seasonal patterns associated with each category of event. For instance, we would expect to see seasonal warm weather cycles for Tornados and Hurricanes.
Storm intensity measures are included in the detail such as wind speed, diameter of hail, category of tornado and hurricane, etc. It would be intersting to understand if there are patterns and trends in storm intensity over time that would suggest a need for review of building standards, evacuation protocols, etc. i.e. better planning for the future.
The effect of inflation on storm damage cost estimates should be factored in. Doing so would make it comparisons of economic damage estimates over time more appropriate.
A careful mapping of raw storm events to the 48 categories listed in Directive 10-1605. More categories would be introduced allowing for finer comparisons of damage estimates and casualties, for instance.
A distinction could be made between property damage and crop damage. Administrators in coastal urban areas would have less interest in crop damage estimates, while mid-west farming communities would have interest in both.
A distinction could be made between injuries and fatalities. For instance, are there important patterns over time or by geography in the rates of injury and death?
The data for this analysis stops in 2011. Four more years of data should soon be available so that patterns we see if the increase in frequency of FLOOD events over time, for example, still stands.
sessionInfo()## R version 3.2.3 (2015-12-10)
## Platform: x86_64-w64-mingw32/x64 (64-bit)
## Running under: Windows 10 x64 (build 10586)
##
## locale:
## [1] LC_COLLATE=English_United States.1252
## [2] LC_CTYPE=English_United States.1252
## [3] LC_MONETARY=English_United States.1252
## [4] LC_NUMERIC=C
## [5] LC_TIME=English_United States.1252
##
## attached base packages:
## [1] grid stats graphics grDevices utils datasets methods
## [8] base
##
## other attached packages:
## [1] gridExtra_2.0.0 scales_0.3.0 ggplot2_2.0.0 knitr_1.12.3
## [5] lubridate_1.5.0 dplyr_0.4.3
##
## loaded via a namespace (and not attached):
## [1] Rcpp_0.12.3 magrittr_1.5 munsell_0.4.2 colorspace_1.2-6
## [5] R6_2.1.2 stringr_1.0.0 highr_0.5.1 plyr_1.8.3
## [9] tools_3.2.3 parallel_3.2.3 gtable_0.1.2 DBI_0.3.1
## [13] htmltools_0.3 yaml_2.1.13 lazyeval_0.1.10 assertthat_0.1
## [17] digest_0.6.9 formatR_1.2.1 codetools_0.2-14 evaluate_0.8
## [21] rmarkdown_0.9.2 labeling_0.3 stringi_1.0-1