this is the main branch
Part 2
## Warning: package 'purrr' was built under R version 4.5.2## Warning: package 'dplyr' was built under R version 4.5.2## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.2.1 ✔ readr 2.1.6
## ✔ forcats 1.0.0 ✔ stringr 1.5.1
## ✔ ggplot2 4.0.1 ✔ tibble 3.3.0
## ✔ lubridate 1.9.4 ✔ tidyr 1.3.1
## ✔ purrr 1.2.2
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors##
## Attaching package: 'rvest'
##
## The following object is masked from 'package:readr':
##
## guess_encodinglibrary(dplyr)
library(stringr)
communityareas<-read_html("https://en.wikipedia.org/wiki/Community_areas_of_Chicago") # loads the html file
#EXTRACT TABLE AS DATAFRAME
CAtable <- communityareas %>%
html_element("table") %>%
html_table()
head(CAtable)## # A tibble: 6 × 7
## No. Name Population `Area[9]` `Area[9]` Density Density
## <chr> <chr> <chr> <chr> <chr> <chr> <chr>
## 1 No. Name .mw-parser-output .n… (sq mi.) (km2) (/sq m… (/km2)
## 2 01 Rogers Park 54,173 1.84 4.77 29,441… 11,367…
## 3 02 West Ridge 78,373 3.53 9.14 22,201… 8,572.…
## 4 03 Uptown 56,344 2.32 6.01 24,286… 9,376.9
## 5 04 Lincoln Square 41,651 2.56 6.63 16,269… 6,281.…
## 6 05 North Center 36,014 2.05 5.31 17,567… 6,782.…FinalCAtable<- CAtable[-c(1,79),] %>% # removes rows 1 & 79 from original
select(No.,Name,Population) %>% # only keeps the number name and population
mutate(
Population = str_remove_all(Population,","),
Population = as.numeric(Population) # removes comma and makes population numeric
)
FinalCAtable %>% slice(24) # West Town## # A tibble: 1 × 3
## No. Name Population
## <chr> <chr> <dbl>
## 1 24 West Town 88164FinalCAtable %>% # Arranges population high to low then gets top 5
arrange(desc(Population)) %>%
head(5)## # A tibble: 5 × 3
## No. Name Population
## <chr> <chr> <dbl>
## 1 08 Near North Side 107331
## 2 06 Lake View 102827
## 3 25 Austin 97452
## 4 24 West Town 88164
## 5 02 West Ridge 78373library(readr)
bookhtml <- read_html("https://books.toscrape.com/")
eachbook <- bookhtml |> # each book section
html_elements("article.product_pod")
eachbook## {xml_nodeset (20)}
## [1] <article class="product_pod"><div class="image_container">\n ...
## [2] <article class="product_pod"><div class="image_container">\n ...
## [3] <article class="product_pod"><div class="image_container">\n ...
## [4] <article class="product_pod"><div class="image_container">\n ...
## [5] <article class="product_pod"><div class="image_container">\n ...
## [6] <article class="product_pod"><div class="image_container">\n ...
## [7] <article class="product_pod"><div class="image_container">\n ...
## [8] <article class="product_pod"><div class="image_container">\n ...
## [9] <article class="product_pod"><div class="image_container">\n ...
## [10] <article class="product_pod"><div class="image_container">\n ...
## [11] <article class="product_pod"><div class="image_container">\n ...
## [12] <article class="product_pod"><div class="image_container">\n ...
## [13] <article class="product_pod"><div class="image_container">\n ...
## [14] <article class="product_pod"><div class="image_container">\n ...
## [15] <article class="product_pod"><div class="image_container">\n ...
## [16] <article class="product_pod"><div class="image_container">\n ...
## [17] <article class="product_pod"><div class="image_container">\n ...
## [18] <article class="product_pod"><div class="image_container">\n ...
## [19] <article class="product_pod"><div class="image_container">\n ...
## [20] <article class="product_pod"><div class="image_container">\n ...## [1] "A Light in the ..."
## [2] "Tipping the Velvet"
## [3] "Soumission"
## [4] "Sharp Objects"
## [5] "Sapiens: A Brief History ..."
## [6] "The Requiem Red"
## [7] "The Dirty Little Secrets ..."
## [8] "The Coming Woman: A ..."
## [9] "The Boys in the ..."
## [10] "The Black Maria"
## [11] "Starving Hearts (Triangular Trade ..."
## [12] "Shakespeare's Sonnets"
## [13] "Set Me Free"
## [14] "Scott Pilgrim's Precious Little ..."
## [15] "Rip it Up and ..."
## [16] "Our Band Could Be ..."
## [17] "Olio"
## [18] "Mesaerion: The Best Science ..."
## [19] "Libertarianism for Beginners"
## [20] "It's Only the Himalayas"## [1] "£51.77" "£53.74" "£50.10" "£47.82" "£54.23" "£22.65" "£33.34" "£17.93"
## [9] "£22.60" "£52.15" "£13.99" "£20.66" "£17.46" "£52.29" "£35.02" "£57.25"
## [17] "£23.88" "£37.59" "£51.33" "£45.17"availability = eachbook |> #availility of each book
html_element("p.instock.availability") |>
html_text2()
availability## [1] "In stock" "In stock" "In stock" "In stock" "In stock" "In stock"
## [7] "In stock" "In stock" "In stock" "In stock" "In stock" "In stock"
## [13] "In stock" "In stock" "In stock" "In stock" "In stock" "In stock"
## [19] "In stock" "In stock"bookstoscrape <- tibble( # creates table
title,
price,
availability
)
cleanbookstoscrape <-bookstoscrape %>% # cleans table and makes price numeric
mutate(
price = str_remove_all(price,"£"),
price = as.numeric(price)
)
print(cleanbookstoscrape)## # A tibble: 20 × 3
## title price availability
## <chr> <dbl> <chr>
## 1 A Light in the ... 51.8 In stock
## 2 Tipping the Velvet 53.7 In stock
## 3 Soumission 50.1 In stock
## 4 Sharp Objects 47.8 In stock
## 5 Sapiens: A Brief History ... 54.2 In stock
## 6 The Requiem Red 22.6 In stock
## 7 The Dirty Little Secrets ... 33.3 In stock
## 8 The Coming Woman: A ... 17.9 In stock
## 9 The Boys in the ... 22.6 In stock
## 10 The Black Maria 52.2 In stock
## 11 Starving Hearts (Triangular Trade ... 14.0 In stock
## 12 Shakespeare's Sonnets 20.7 In stock
## 13 Set Me Free 17.5 In stock
## 14 Scott Pilgrim's Precious Little ... 52.3 In stock
## 15 Rip it Up and ... 35.0 In stock
## 16 Our Band Could Be ... 57.2 In stock
## 17 Olio 23.9 In stock
## 18 Mesaerion: The Best Science ... 37.6 In stock
## 19 Libertarianism for Beginners 51.3 In stock
## 20 It's Only the Himalayas 45.2 In stockPart Three
##
## Attaching package: 'jsonlite'## The following object is masked from 'package:purrr':
##
## flatten# answer to part 3 question 1
url <- "https://data.cityofchicago.org/resource/v6vf-nfxy.json"
params <- list( #creates a filter that will go through API
community_area = "24",
`$where` = "created_date >= '2026-01-01T00:00:00'",
`$order` = "created_date DESC",
`$limit` = 1000
)
res <- GET(url, query = params)
#question 2
dat <- fromJSON(content(res, "text")) # dataframe created fromJSON
head(dat)## sr_number sr_type sr_short_code
## 1 SR26-02113503 Water On Street Complaint AAE
## 2 SR26-02113467 Rodent Baiting/Rat Complaint SGA
## 3 SR26-02113450 Traffic Signal Out Complaint SFB
## 4 SR26-02113410 Sign Repair Request - All Other Signs PCE
## 5 SR26-02113379 Sidewalk Café/Outdoor Dining Complaint CAFE
## 6 SR26-02113355 311 INFORMATION ONLY CALL 311IOC
## owner_department status origin
## 1 DWM - Department of Water Management Open Mobile Device
## 2 Streets and Sanitation Open Mobile Device
## 3 CDOT - Department of Transportation Open Internet
## 4 CDOT - Department of Transportation Open Mobile Device
## 5 BACP - Business Affairs and Consumer Protection Open Mobile Device
## 6 311 City Services Completed Phone Call
## created_date last_modified_date street_address
## 1 2026-10-18T16:54:37.000 2026-10-18T16:54:40.000 1959 W HURON ST
## 2 2026-10-18T16:43:45.000 2026-10-18T16:43:47.000 1702 W WABANSIA AVE
## 3 2026-10-18T16:39:14.000 2026-10-18T16:39:18.000 1363 N ASHLAND AVE
## 4 2026-10-18T16:29:00.000 2026-10-18T16:29:04.000 1087 N PAULINA ST
## 5 2026-10-18T16:22:27.000 2026-10-18T16:22:31.000 1702 N DAMEN AVE
## 6 2026-10-18T16:17:14.000 2026-10-18T16:33:00.000 2111 W Lexington ST
## street_number street_direction street_name street_type duplicate
## 1 1959 W HURON ST FALSE
## 2 1702 W WABANSIA AVE FALSE
## 3 1363 N ASHLAND AVE FALSE
## 4 1087 N PAULINA ST FALSE
## 5 1702 N DAMEN AVE FALSE
## 6 2111 W Lexington ST FALSE
## legacy_record community_area ward electrical_district electricity_grid
## 1 FALSE 24 36 5 L021
## 2 FALSE 24 32 5 L018
## 3 FALSE 24 1 5 M019
## 4 FALSE 24 1 5 L020
## 5 FALSE 24 32 5 K018
## 6 FALSE 24 1 5 L020
## police_sector police_district police_beat precinct created_hour
## 1 2 12 1221 27 16
## 2 3 14 1433 25 16
## 3 3 14 1433 21 16
## 4 1 12 1213 22 16
## 5 3 14 1434 27 16
## 6 1 12 1213 24 16
## created_day_of_week created_month x_coordinate y_coordinate
## 1 1 10 1162994.1385407883 1904605.5039792012
## 2 1 10 1164677.5544678858 1911437.1459347906
## 3 1 10 1165582.9609621807 1909361.485400256
## 4 1 10 1164928.7112666995 1907631.2391312786
## 5 1 10 1162671.5701363974 1911382.3709340198
## 6 1 10 1164814.74398591 1906367.6358646809
## latitude longitude location.latitude location.longitude
## 1 41.893893000941 -87.676830000002 41.893893000941 -87.676830000002
## 2 41.912604000941 -87.670453500002 41.912604000941 -87.670453500002
## 3 41.906889000941 -87.667186500002 41.906889000941 -87.667186500002
## 4 41.902155000941 -87.669639000002 41.902155000941 -87.669639000002
## 5 41.912496000941 -87.677824500002 41.912496000941 -87.677824500002
## 6 41.898690000941 -87.670093500002 41.898690000941 -87.670093500002
## location.human_address city state zip_code
## 1 {"address": "", "city": "", "state": "", "zip": ""} <NA> <NA> <NA>
## 2 {"address": "", "city": "", "state": "", "zip": ""} <NA> <NA> <NA>
## 3 {"address": "", "city": "", "state": "", "zip": ""} Chicago Illinois 60642
## 4 {"address": "", "city": "", "state": "", "zip": ""} <NA> <NA> <NA>
## 5 {"address": "", "city": "", "state": "", "zip": ""} <NA> <NA> <NA>
## 6 {"address": "", "city": "", "state": "", "zip": ""} Chicago Illinois 60612
## created_department closed_date parent_sr_number
## 1 <NA> <NA> <NA>
## 2 <NA> <NA> <NA>
## 3 <NA> <NA> <NA>
## 4 <NA> <NA> <NA>
## 5 <NA> <NA> <NA>
## 6 311 City Services 2026-10-18T16:17:15.000 <NA>## [1] 1000## [1] "sr_number" "sr_type" "sr_short_code"
## [4] "owner_department" "status" "origin"
## [7] "created_date" "last_modified_date" "street_address"
## [10] "street_number" "street_direction" "street_name"
## [13] "street_type" "duplicate" "legacy_record"
## [16] "community_area" "ward" "electrical_district"
## [19] "electricity_grid" "police_sector" "police_district"
## [22] "police_beat" "precinct" "created_hour"
## [25] "created_day_of_week" "created_month" "x_coordinate"
## [28] "y_coordinate" "latitude" "longitude"
## [31] "location" "city" "state"
## [34] "zip_code" "created_department" "closed_date"
## [37] "parent_sr_number"dat %>%
count(sr_type, sort = TRUE)%>% # uses data frame and sorts by sr type from descending order then gives top 10. Answer to 3
head(10)## sr_type n
## 1 Graffiti Removal Request 201
## 2 Rodent Baiting/Rat Complaint 54
## 3 Finance Parking Code Enforcement Review 46
## 4 Yard Waste Pick-Up Request 46
## 5 Pothole in Street Complaint 36
## 6 Sign Repair Request - All Other Signs 34
## 7 Street Light Out Complaint 31
## 8 Tree Emergency 30
## 9 Abandoned Vehicle Complaint 28
## 10 Garbage Cart Maintenance 27Part 3 Question 4: When looking at this data it can be seen that a lot of requests are to remove graffiti in West Town so that may be prominent. One thing this data can’t tell me is how long this graffiti removal can take since there can be some that take a while to remove depending on size.
Part Four:
## Warning: package 'tidycensus' was built under R version 4.5.2#used from class example in order to find variables in r
y24 <- load_variables(2024, "acs5", cache = FALSE)
print(y24) ## # A tibble: 28,437 × 4
## name label concept geography
## <chr> <chr> <chr> <chr>
## 1 B01001A_001 Estimate!!Total: Sex by Age (W… tract
## 2 B01001A_002 Estimate!!Total:!!Male: Sex by Age (W… tract
## 3 B01001A_003 Estimate!!Total:!!Male:!!Under 5 years Sex by Age (W… tract
## 4 B01001A_004 Estimate!!Total:!!Male:!!5 to 9 years Sex by Age (W… tract
## 5 B01001A_005 Estimate!!Total:!!Male:!!10 to 14 years Sex by Age (W… tract
## 6 B01001A_006 Estimate!!Total:!!Male:!!15 to 17 years Sex by Age (W… tract
## 7 B01001A_007 Estimate!!Total:!!Male:!!18 and 19 years Sex by Age (W… tract
## 8 B01001A_008 Estimate!!Total:!!Male:!!20 to 24 years Sex by Age (W… tract
## 9 B01001A_009 Estimate!!Total:!!Male:!!25 to 29 years Sex by Age (W… tract
## 10 B01001A_010 Estimate!!Total:!!Male:!!30 to 34 years Sex by Age (W… tract
## # ℹ 28,427 more rows• Gross Rent as a Percentage of Household Income, “Total”: B25070_001E • Gross Rent as a Percentage of Household Income, “50.0 percent or more”:B25070_010E • Median Household Income: B19013_001E
# step 2 creates dataset for Ohio with all 3 variables for all Ohio counties
Ohioincome <- get_acs(geography = "tract",
variables = c(
percent_rent_total = "B25070_001", # gets all 3 variables
fiftypercent_rent="B25070_010",
median_householdinc= "B19013_001" ),
state = "OH", ,
year = 2024,
survey = "acs5") ## Getting data from the 2020-2024 5-year ACS## # A tibble: 6 × 5
## GEOID NAME variable estimate moe
## <chr> <chr> <chr> <dbl> <dbl>
## 1 39001770100 Census Tract 7701; Adams County; Ohio median_house… 60054 7276
## 2 39001770100 Census Tract 7701; Adams County; Ohio percent_rent… 564 139
## 3 39001770100 Census Tract 7701; Adams County; Ohio fiftypercent… 66 48
## 4 39001770200 Census Tract 7702; Adams County; Ohio median_house… 53728 6145
## 5 39001770200 Census Tract 7702; Adams County; Ohio percent_rent… 448 135
## 6 39001770200 Census Tract 7702; Adams County; Ohio fiftypercent… 69 37##
## Attaching package: 'janitor'## The following objects are masked from 'package:stats':
##
## chisq.test, fisher.test# step 3
wideOhioIncome <-Ohioincome %>% #uses Ohioincome dataset and gets rid of moe
select(-moe)%>%
separate_wider_delim( #used AI here to figure out how to seperate the name into different columns
cols = NAME,
delim = "; ",
names = c("census_tract", "county", "state")
) %>%
pivot_wider( #makes it wider by giving each variable its own column
id_cols = c(GEOID,census_tract,county,state),
names_from = variable,
values_from = c(estimate),
) %>%
clean_names() %>%
#step 3.2
mutate( # adds variable to table
severe_burden_rate = fiftypercent_rent/ percent_rent_total
) %>%
arrange(desc(severe_burden_rate))
print(wideOhioIncome)## # A tibble: 3,168 × 8
## geoid census_tract county state median_householdinc percent_rent_total
## <chr> <chr> <chr> <chr> <dbl> <dbl>
## 1 39017010104 Census Tract… Butle… Ohio NA 23
## 2 39017011006 Census Tract… Butle… Ohio 112632 20
## 3 39025041506 Census Tract… Clerm… Ohio 98179 11
## 4 39041011764 Census Tract… Delaw… Ohio 177500 6
## 5 39041011766 Census Tract… Delaw… Ohio 127500 123
## 6 39035120801 Census Tract… Cuyah… Ohio 21839 503
## 7 39035136105 Census Tract… Cuyah… Ohio 131667 68
## 8 39035154603 Census Tract… Cuyah… Ohio 55760 331
## 9 39095008205 Census Tract… Lucas… Ohio 142791 35
## 10 39095009001 Census Tract… Lucas… Ohio 158021 77
## # ℹ 3,158 more rows
## # ℹ 2 more variables: fiftypercent_rent <dbl>, severe_burden_rate <dbl>STEP Four
#gets median and mode for severe burden rate
wideOhioIncome %>%
summarise(
mean_inc = mean(severe_burden_rate, na.rm = TRUE),
median_inc = median(severe_burden_rate, na.rm = TRUE)
)## # A tibble: 1 × 2
## mean_inc median_inc
## <dbl> <dbl>
## 1 0.199 0.182#2
countyrates <- wideOhioIncome %>% #groups by county gets mean and arranges them
group_by(county) %>%
summarise(avg_rate = mean(severe_burden_rate, na.rm = TRUE), .groups = "drop") %>%
arrange(desc(avg_rate))
#top 5 highest rates
head(countyrates, 5)## # A tibble: 5 × 2
## county avg_rate
## <chr> <dbl>
## 1 Athens County 0.270
## 2 Pike County 0.252
## 3 Scioto County 0.245
## 4 Butler County 0.245
## 5 Jackson County 0.244## # A tibble: 5 × 2
## county avg_rate
## <chr> <dbl>
## 1 Union County 0.0939
## 2 Carroll County 0.0914
## 3 Holmes County 0.0826
## 4 Mercer County 0.0817
## 5 Noble County 0.0336Step 5
library(ggplot2)
#creates scatterplot
ggplot(data = wideOhioIncome, aes(x = median_householdinc, y = severe_burden_rate)) +
geom_point(alpha = 0.5, color = "blue") +
geom_smooth(method = "lm", color = "red", se = FALSE) +
labs(
title = "Median Household Income vs. Severe Burden Rate",
subtitle = "Ohio Census Tracts",
x = "Median Household Income ($)",
y = "Severe Burden Rate"
) +
theme_minimal()## `geom_smooth()` using formula = 'y ~ x'## Warning: Removed 40 rows containing non-finite outside the scale range
## (`stat_smooth()`).## Warning: Removed 40 rows containing missing values or values outside the scale range
## (`geom_point()`).
#creates bar graph
top_5 <- head(countyrates, 5)
low_5 <- tail(countyrates, 5)
#getting them to be percentages
countybargraph <- bind_rows(top_5, low_5) %>%
mutate(avg_ratepercent = avg_rate * 100)
ggplot(countybargraph, aes(x = avg_ratepercent, y = reorder(county, avg_ratepercent))) + # orders from high to low
geom_col( fill = "blue") +
labs(
title = "Top 5 and Bottom 5 Counties by Severe Housing Burden Rate",
subtitle = "Ohio Counties",
x = "Average Severe Housing Burden Rate (%)",
y = "County"
) +
theme_minimal()
Step 6:
1. What relationship do you see between income and severe rent burden in Ohio, and what might explain it?
The relationship I see is that as median household income increases the severe burden rate decreases which makes sense. If the income increases, they are more likely to afford their rent compared to others who have lower income who make enough to cover rent but are limited with other things.
- What are two limitations of using county-level ACS estimates to decide where to target rental assistance?
One limitation is that at county-level there could be drastic ranges of income in a single county. A really wealthy neighborhood vs an impoverished neighborhood which can skew the numbers. Another limitation is that it is every 5 years and a lot can happen in that time span that can change the circumstances at each county.
Part 5:
- The 311 dataset includes the street address and coordinates of each request. Suppose you wanted to publish a map of West Town requests on your GitHub Page. Identify one privacy risk, then explain how you would apply one de-identification technique before publishing.
Answer: One privacy risk is that when making a map, it will show what area exactly in West Town is having certain issues. If there is a rodent problem, it exposes people who live nearby when most likely they would want to not have that type of issue made public and just have it resolved. One technique I would apply is aggregation. I would use summary statistics and get total counts for issues instead of just doing each individual requests. I would also just get of exact street address as well.
- Bias and data quality. 311 data shows where people report problems, which is not always where problems exist. Explain how this could mislead a city manager who assigns repair crews by request counts, and describe one way to check for or reduce that bias.
Answer:This could mislead a city manager because they would send crews to the area where people only report the problems and not other areas where problems also exist. This would mean the crews would be sent only where requested and leaves other areas who may not report to be unserved or served to a lesser degree which is not appropriate. One way to reduce this bias is to be proactive and find which areas report less. It may not be because there is no issues but less reporting. Going into these areas and informing residents about how to report but also checking if there are issues.
- Scraping responsibly. Using the criteria discussed in Week 5, explain whether the scraping you did in Part II was appropriate. Then describe one civic data situation where you should not scrape without further review.
Answer: I would say the scraping I did in part 2 was appropriate. The data was on Wikipedia which was public, it was not about private individuals and it was also factual. A civic data situation where it would not be appropriate is when it is not public and can also about an individual. It can be a government website that collects an individuals data such as name, address and phone number. Scraping that is non-consensual and most likely is not data that is available for the public.
AI Use Statement: I used Gemini AI in areas where I may have gotten errors when running chunks in order for it to run efficiently or to get the correct outcome. There may have been a line missing that I needed. I also had a problem at one point when merging branches so I asked for the steps that I needed to take in order to resolve issue.