Advanced GIS, Week 8

File organization is very important for to keep track of data and your analysis.

We’ll all work in these folders as you learn so that we can follow my lessons more easily.

• Download the main data folder and save it in your advanced-gis folder.

From R for Data Science Hadley Wickham & Garrett Grolemund

• R: a programming language and software environment for statistical computing and graphics
• RStudio: an application that helps you write in R in a user-friendly way
• R script: an executable document with instructions in R to complete a task (it’s like a recipe!).
• function: a set of statements organized together to perform a specific task.
• Base R: the functions that come standard with your R installation.
• R package: a collection of functions created by the community that other R users can use.

I use Quarto to make slides for this course.

• They contains text and executable R code together

• Text in a box is a code chunk - something you can execute in R

• The output from the code chunk displays below the box

# this is a gray box
print("this is the output from the code chunk")

[1] "this is the output from the code chunk"

• You can follow along with all of the lectures in R Studio on your computer.

• R scripts sometimes run slightly differently on different machines
  (different operating systems, R versions, package versions, etc)
• You can google R, try it! (“r import csv”)
• Other people have probably had the same problem as you or asked the same question
  • Stack Overflow
    
  • Google your error message
• Style matters! Make it readable for future you, and for your colleagues
  • Indent for readability
  • Include comments explaining what you are doing/thinking
  • Include sources in your script
• Learn R shortcuts (cheatsheet)

• Part 1 R: Data Manipulation for Mapping
  • due October 25
• Part 2 R: tidycensus + revisions to part 1
  • due November 1

due October 25

• Deliverable 1 R script to process student poverty data for New York school districts in 2018, save it to your computer, and create summary statistics about student poverty in New York.
• Deliverable 1.1 (optional) R script to join student poverty data to school district shapefile
• Deliverable 2 Choropleth map of school districts in New York, styled by student poevrty rate
• Deliverable 3 R script to process student poverty data for New York school districts in 2019, save it to your computer, and create summary statistics about student poverty in New York.

R is installed on all of the computers in this lab. You can install on your computer if you want to work from home.

• First, install R

• Second, install RStudio

Create a New R Project in your existing folder called main_data

The Console is:

• a calculator: Type code, press enter, the answer displays below.
• terminal: Warnings and errors are displayed in the console

Type into your console, and then press enter:

4 + 5

[1] 9

14 - seven

Use the Source Pane to write R scripts.

• scripts are like a recipe, a list of instructions for how to process your data

• In R Studio, select File > New File > R Script
  • Save the script in your main_data/scripts/data_exploration folder as ny_process_school_district_data_2018.R
  • At the top, type “# this is my first script” (make sure you include the hash #)
  • Create a variable called seven
  • Type 14 - seven
  • Highlight both lines with your cursor and click Run in the upper-right of your Source pane

# this is my first script

seven <- 7
14 - seven

[1] 7

The Files window is like file explorer. You can see all of the files on your computer

The Packages window lists the packages you have installed and provides a user interface to search for other packages and install them.

Packages are collections of functions and datasets developed by the R community to expand the things you can do in R.

• Some, like the tidyverse, have become the backbone of analysis in R and we use it all the time
• Install the tidyverse by typing into your console:

install.packages(tidyverse)

• You only need to install a package onto your computer once
• But you need to add any script you use in a script at the top so R knows you are using it

The Environment shows all of the objects that you have in your workspace

• If you are following along, you should have at least 1 object in your Environment: the variable “seven”.

• We will import a csv of every school district in New York that is in your main_data/raw/school_district folder

In your script:

• Delete your practice equations
• At the top of you script, describe what you will do in this script (see below)
• Load the tidyverse package into your environment
• “read in” your csv into R Studio

## Process 2018 enrollment data for every school district in the country

library(tidyverse)

## import the education dataset for 2019 using read_csv from the readr package of the tidyverse
# source = https://www.census.gov/data/datasets/2018/demo/saipe/2018-school-districts.html
raw_sd_poverty_18 <- read_csv("raw/school_district/ny_student_poverty_2018.csv")

The specification of the list of folders to get to a file on your computer is called a path.

• Absolute path: starts at the root folder of the computer

• Relative path: starts at a given folder and provides the path starting from that folder

• Windows file paths use forward slashes (R doesn’t like them)
• If you have issues with an absolute filepath, use the file.path() function

Data tables are called dataframes in R.

• When you read a table into your R Environment, all of changes that you make to the dataframe are not saved to your computer until you save the dataframe to your computer (in R we say write it out).

Let’s explore our first data frame:

• To view your dataframe, click on the name in your Environment pane

Type glimpse(raw_sd_poverty_18) in the console to look at a snapshot of your dataframe

• Numeric
  • Integers (whole numbers)
  • Doubles (fractions)
• Character (string)
• Logical (boolean) - TRUE or FALSE

We’ll learn five functions that are the backbone of data transformation in R

• mutate() - create or redefine a variable (column)
• select() - subset variables (columns) by their names
• filter() - subset observations (rows) by their values
• rename() - rename the variables
• summarise() - calculate summary statistics

• Create new data frame from raw_stpov18
• New column that equals the student poverty rate

stpov18 <- raw_sd_poverty_18 %>%
  mutate(stud_pov_rate = Estimated_pop_age_5_17_in_poverty/Estimated_pop_age_5_17)

• year = “2018”

stpov18 <- raw_sd_poverty_18 %>%
  mutate(stud_pov_rate = Estimated_pop_age_5_17_in_poverty/Estimated_pop_age_5_17,
         year = "2018")

• subset columns by their names

Our student poverty table has 9 columns, we may not need all of those.

Type names(stpov18) in your console to see column names

stpov18 <- raw_sd_poverty_18 %>%
  mutate(stud_pov_rate = Estimated_pop_age_5_17_in_poverty/Estimated_pop_age_5_17,
         year = "2018") %>%
  select(Postal, district_id, Name, Estimated_total_pop, 
         Estimated_pop_age_5_17, Estimated_pop_age_5_17_in_poverty, stud_pov_rate, year)

rename your columns * new_name = old_name * no spaces in variable names

stpov18 <- raw_sd_poverty_18 %>%
  mutate(stud_pov_rate = Estimated_pop_age_5_17_in_poverty/Estimated_pop_age_5_17,
         year = "2018") %>%
  select(Postal, district_id, Name, Estimated_total_pop, 
         Estimated_pop_age_5_17, Estimated_pop_age_5_17_in_poverty, stud_pov_rate, year) %>%
  rename(id = district_id,
         district = Name,
         tpop = Estimated_total_pop,
         stpop = Estimated_pop_age_5_17,
         stpov = Estimated_pop_age_5_17_in_poverty,
         stpovrate = stud_pov_rate)

• subset observations (rows) by their values

• Create new dataframe without districts with fewer than 100 children

stpov18_no_low_enroll <- stpov18 %>%
  filter(stpop >= 100)

Save this processed data frame dataframe to your computer

# write out processed student poverty data for 2018
write_csv(stpov18, "processed/school_district/ny_student_poverty_rate_2018.csv")

Now that we have our analysis dataset, it’s time to use it to learn more about poverty in New York state. We’ll use descriptive statistics and visualization to interpret the data.

• Summary statistics for each variable with the summary() function
• A very simple scatterplot & histogram to understand the shape of our data
• Calculate our own descriptive statistics for the state with the summarise() function

• type into your console

A histogram is a chart that shows the distribution of your data.

• The height of each bar indicates how many district’s poverty rate is within that range.
• To create a histogram use the hist() function, with format:
  • hist(dataframe_name$column_name)
  • $ indicates a column name

• We can use the summarise() function to create our own statistics to help answer our questions.
  • we’ll create a new dataframe that is a single row summarizing our dataframe
  • useful arguments within summarize:
    • mean(), median(), min(), max(), sum(), n()
    • n() returns the number of rows

Save summary stats dataframe to your computer

# write out student poverty summary stats for 2018
write_csv(ny_pov_stats, "output/school_district/ny_student_poverty_rate_statistics_2018.csv")

Neaten up the script for processing the student-age poverty rate for New York school districts in 2018. It should include everything we have done above:

• One sentence description at the top of the purpose of the script (commented out)
• Load the tidyverse package needed to run the script
• Read in the raw school-age poverty rate data
• Transform it with the mutate(), rename(), select()
• Create 5 additional dataframes using the filter() function
• Write out the processed stpov18 dataframe
• Quick summary statistics with summary()
• Quick histogram with hist()
• Bespoke summary statistics with summarise()
• Write out summary statistics dataframe

Upload your script to CANVAS

Join poverty data to school district shapefile and make a choropleth map of student poverty rate in QGIS or ArcMap.

Step 1: Two options to join the data:

1. Use QGIS or ArcMap to join school district poverty rate data to a school district shapefile
2. Use R to join school district poverty rate data to a school district shapefile (see option 2 slides for instructions)

Step 2: Create choropleth map of New York school districts, colored by student-age poverty rate in 2018

• make the map nice. include title, legend, carefully chosen bins for the student-age poverty rate

Upload your map (and optionally, your script to join the data to the shapefile)

• Create a new script called ny_school_districts_poverty_shape_join.R and save it in main_data/scripts/data_processing
• Install sf the best package for handling spatial data in R
  • In the packages tab, click install, search for sf, and install in default location
• Read in your processed 2018 student poverty dataset
• The the sf function st_read() to read in the school district shapefile
• Use the left_join() function to join the poverty rate data to the shapefile
  • for any join you have to find the common key (or common, unique column that is in both datasets)
  • in this case, they have a common column with different names GEOID and id

One of the magic aspects of scripts is you can reuse them! You are going to process the 2019 poverty rate data by making a copy of your 2018 script and slightly adjusting it to work with the 2019 data.

• Save a copy of your ny_process_school_district_data_2018.R script as ny_process_school_district_data_2019.R in the main_data/scripts/data_exploration folder.
• You’ll find the raw 2019 student-age poverty data in the same folder as the 2018 data.
• Adjust your 2019 script so that it imports the 2019 data (hint, change all the “18”s to “19”s)
• There is one column that is named slightly differently, so you’ll need to use the rename() function to change it.
• Use this script to do exactly the same processing steps to the 2019 data as you did for 2018 data, including saving the processed data and the summary stats to the same folders
• There is an additional data file in the raw/school_district folder that has enrollment data for every school district in the country (full_data_19_geo_exc.csv). Optionally, read it in and explore it for more practice. See Optional Deliverable 3.1 for instructions.

Upload your script to CANVAS

Process the 2019 school district enrollment dataset for 2019 and join it to the poverty dataframe.

In your ny_process_school_district_data_2019.R script:

• Use read_csv() to read in the full_data_19_geo_exc.csv dataset of enrollment data for every school district in the country.
• This dataset has a lot of fields; create a dataframe with the following variables:
  • NCESID, NAME, County, dEnroll_district, dWhite, dHispanic, dBlack, dAsian_PI
  • that is the id field, district name, and enrollment by race/ethnicity
• Create the following variables:
  • percent_latinx, percent_black, percent_white, percent_asian
  • convert NCESID to numeric type (mutate(NCESID = as.numeric(NCESID)))
• Create a new dataframe called ny_pov_enroll from your 2019 ny poverty dataframe
  • Use left_join() to join the enrollment data to the ny poverty data
    • Notice that the left join keeps all of the rows in the first dataset, and add columns from the second dataframe that match
    • There will be 5 districts from the enrollment dataframe that don’t join properly (don’t worry about them for now)

Upload this script to CANVAS for .5 points for processing the enrollment data, and .5 points for joining the dataframes