Connecting to FEMA’s API

One stop guide to collecting summaries of natural disasters

Author

Ben

Overview of FEMA’s API Services

FEMA (Federal Emergency Management Agency), like many government agencies has open API’s and offers easy and free access to data. To connect to FEMA’s API all you need is an endpoint, a desired data set and any possible filters.

Like many other API services, they ask that users make less than 1000 “GET” requests per hour. Not only this but FEMA only allows up too 10000 rows to be pulled per request. Since there are not too many natural disasters to be studied this is not a problem.

I have decided to use FEMA’s API to pull summaries of Natural disasters ranging back to the 1970’s from all 50 states.

Using this tutorial you will be able to get summaries of natural disasters as well as how to use filtering to make the most out of each request.

To learn about the API follow this link and what logical operators can be used to filter follow this link and find the “logical operators” section to use the filtering capabilities: openFEMA API

Tutorial

First we need to load our packages!

library(tidyverse) # All the tidy things
library(jsonlite)  # Converting json data into data frames
library(magrittr)  # Tis not a pipe
library(httr)      # Interacting with HTTP verbs
library(DT)        # For all those pretty Data Tables

You will use tidyverse grammar for function building as well as interacting with the API. The jsonlite package allows you to convert JSON files into our normal columns and rows without needing a key. The magrittr package will allow us to use the pipe operator (%>%) in places where tidyverse grammar is unusable. The httr package will allow us to make “GET” requests directly from Rstudio.

fema_url <- 'https://www.fema.gov/api/open'
fema_dataset <- '/v2/DisasterDeclarationsSummaries?$select='
fields <- paste('femaDeclarationString','state','declarationType','declarationDate',
                'fyDeclared','incidentType','declarationTitle','incidentBeginDate',
                'incidentEndDate','designatedIncidentTypes', 'lastRefresh', sep = ',')
parameters <- '$top=10000'

fema_api_url <- 
  paste(fema_url,fema_dataset,fields,sep = '')

fema_api_url_response <- 
  fema_api_url %>% 
  GET() 

fema_api_data <- 
  fema_api_url_response %>% 
  content(as = "text",
          encoding = "UTF-8") %>% # UTF-8 is the <near> universal standard
  fromJSON()

Using this code we will pull a JSON with metadata as well as a summary for natural disasters. To access this data set and allow us to filter easily, I will include a function that can be hosted in your global environment that will allow you to do this. There are two functions, one that creates a URL, that must be hosted in the global environment. The second is a function with the same inputs as the first function, but it will also interact with the API and pull a table of rows and columns that can then be analyzed.

request_fema_url <- 
  function(filter_column,filter_logical,filter_by, records) {
    fema_url <- 'https://www.fema.gov/api/open' # URL
    fema_dataset <- '/v2/DisasterDeclarationsSummaries?$select=' # Dataset chosed
    fields <- paste('femaDeclarationString','state','declarationType','declarationDate',
                    'fyDeclared','incidentType','declarationTitle','incidentBeginDate',
                    'incidentEndDate','designatedIncidentTypes', 'lastRefresh', sep = ',') 
                    # columns I am including in the dataset
    if(missing(filter_column) | missing(filter_logical) | missing(filter_by))  {
      filter <- ""
    } else {
      filter <- paste("&$filter=",filter_column,"%20",filter_logical,"%20%27",
                      filter_by, '%27', sep = "")    
    }
    # builds our filter string
    # If one of the 3 parameters is missing it will not be added to the URL
    if (missing(records))  {
      num_records <- ""
    } else {
      num_records <- paste("&$top=",records,sep = "")
    }
   # How many records we want to pull (10000 max)
    fema_api_url <- 
      paste(fema_url,fema_dataset,fields,filter,num_records,sep = "")
   # Pastes URL together
  return(fema_api_url)
   # Returns a URL that can be used as a "GET" request
  }

## This function will take the same inputs as the URL creator, but it will now retrieve the      data

# This is our "GET" request
request_fema_url_df <- 
  function(filter_column, filter_logical, filter_by, records) {
    # Start by making use of the function we have already created to generate a
    # URL:
    fema_api_url <- 
      request_fema_url(filter_column, filter_logical, filter_by, records)
    
    # Use the URL to retrieve the JSON data and create the data frame
    fema_data <- 
      fema_api_url %>% 
      GET() %>% 
      content(as = "text",
              encoding = "UTF-8") %>% 
      fromJSON() %>% 
      use_series(DisasterDeclarationsSummaries) 
    # When finished, have the function return the scorecard data frame output.
    return(fema_data)
  }

Now that you have these two functions, what does it look like when we use the ‘request_fema_url_df’ function?

ny_disaster_summary <- # Name of DF
request_fema_url_df(filter_column = 'state', # filter on column state
                    filter_logical = 'eq', # use logical operator for equal
                    filter_by = 'NY', # filter by the state of NY
                    records = '10000')   # pull first 10000 records (max)

datatable(ny_disaster_summary)

Now we with this function we have a summary of all natural disasters that have hit NY. Using the HTML table above you can do a quick data exploration. Using these functions you can easily interact with FEMA’s open API using only tidyverse grammar!

Conclusion

Now with two functions and five inputs you can learn about natural disasters in your state or in others! Enjoy!

--- title: "Connecting to FEMA's API" # Name of your HTML output subtitle: "One stop guide to collecting summaries of natural disasters" author: "Ben" # Author name toc: true # Generates an automatic table of contents. format: # Options related to formatting. html: # Options related to HTML output. code-tools: TRUE # Allow the code tools option showing in the output. embed-resources: TRUE # Embeds all components into a single HTML file. execute: # Options related to the execution of code chunks. warning: FALSE # FALSE: Code chunk warnings are hidden by default. message: FALSE # FALSE: Code chunk messages are hidden by default. echo: TRUE # TRUE: Show all code in the output. --- # Overview of FEMA's API Services FEMA (Federal Emergency Management Agency), like many government agencies has open API's and offers easy and free access to data. To connect to FEMA's API all you need is an endpoint, a desired data set and any possible filters. Like many other API services, they ask that users make less than 1000 "GET" requests per hour. Not only this but FEMA only allows up too 10000 rows to be pulled per request. Since there are not too many natural disasters to be studied this is not a problem. I have decided to use FEMA's API to pull summaries of Natural disasters ranging back to the 1970's from all 50 states. Using this tutorial you will be able to get summaries of natural disasters as well as how to use filtering to make the most out of each request. To learn about the API follow this link and what logical operators can be used to filter follow this link and find the "logical operators" section to use the filtering capabilities: [openFEMA API](https://www.fema.gov/about/openfema/api) ## Tutorial First we need to load our packages! ```{r} #| label: Library Load library(tidyverse) # All the tidy things library(jsonlite) # Converting json data into data frames library(magrittr) # Tis not a pipe library(httr) # Interacting with HTTP verbs library(DT) # For all those pretty Data Tables ``` You will use tidyverse grammar for function building as well as interacting with the API. The jsonlite package allows you to convert JSON files into our normal columns and rows without needing a key. The magrittr package will allow us to use the pipe operator (%\>%) in places where tidyverse grammar is unusable. The httr package will allow us to make "GET" requests directly from Rstudio. ```{r} #| label: Setting the endpoint and and query string fema_url <- 'https://www.fema.gov/api/open' fema_dataset <- '/v2/DisasterDeclarationsSummaries?$select=' fields <- paste('femaDeclarationString','state','declarationType','declarationDate', 'fyDeclared','incidentType','declarationTitle','incidentBeginDate', 'incidentEndDate','designatedIncidentTypes', 'lastRefresh', sep = ',') parameters <- '$top=10000' fema_api_url <- paste(fema_url,fema_dataset,fields,sep = '') fema_api_url_response <- fema_api_url %>% GET() fema_api_data <- fema_api_url_response %>% content(as = "text", encoding = "UTF-8") %>% # UTF-8 is the <near> universal standard fromJSON() ``` Using this code we will pull a JSON with metadata as well as a summary for natural disasters. To access this data set and allow us to filter easily, I will include a function that can be hosted in your global environment that will allow you to do this. There are two functions, one that creates a URL, that must be hosted in the global environment. The second is a function with the same inputs as the first function, but it will also interact with the API and pull a table of rows and columns that can then be analyzed. ```{r} #| label: Function for building URL as well as GET request request_fema_url <- function(filter_column,filter_logical,filter_by, records) { fema_url <- 'https://www.fema.gov/api/open' # URL fema_dataset <- '/v2/DisasterDeclarationsSummaries?$select=' # Dataset chosed fields <- paste('femaDeclarationString','state','declarationType','declarationDate', 'fyDeclared','incidentType','declarationTitle','incidentBeginDate', 'incidentEndDate','designatedIncidentTypes', 'lastRefresh', sep = ',') # columns I am including in the dataset if(missing(filter_column) | missing(filter_logical) | missing(filter_by)) { filter <- "" } else { filter <- paste("&$filter=",filter_column,"%20",filter_logical,"%20%27", filter_by, '%27', sep = "") } # builds our filter string # If one of the 3 parameters is missing it will not be added to the URL if (missing(records)) { num_records <- "" } else { num_records <- paste("&$top=",records,sep = "") } # How many records we want to pull (10000 max) fema_api_url <- paste(fema_url,fema_dataset,fields,filter,num_records,sep = "") # Pastes URL together return(fema_api_url) # Returns a URL that can be used as a "GET" request } ## This function will take the same inputs as the URL creator, but it will now retrieve the data # This is our "GET" request request_fema_url_df <- function(filter_column, filter_logical, filter_by, records) { # Start by making use of the function we have already created to generate a # URL: fema_api_url <- request_fema_url(filter_column, filter_logical, filter_by, records) # Use the URL to retrieve the JSON data and create the data frame fema_data <- fema_api_url %>% GET() %>% content(as = "text", encoding = "UTF-8") %>% fromJSON() %>% use_series(DisasterDeclarationsSummaries) # When finished, have the function return the scorecard data frame output. return(fema_data) } ``` Now that you have these two functions, what does it look like when we use the 'request_fema_url_df' function? ```{r} #| label: Example of Get Request ny_disaster_summary <- # Name of DF request_fema_url_df(filter_column = 'state', # filter on column state filter_logical = 'eq', # use logical operator for equal filter_by = 'NY', # filter by the state of NY records = '10000') # pull first 10000 records (max) datatable(ny_disaster_summary) ``` Now we with this function we have a summary of all natural disasters that have hit NY. Using the HTML table above you can do a quick data exploration. Using these functions you can easily interact with FEMA's open API using only tidyverse grammar! ## Conclusion Now with two functions and five inputs you can learn about natural disasters in your state or in others! Enjoy!