# load the libraries
library(tidyverse)
library(dplyr)
library(plotly) # interactive graphProject 2 - Air quality
DATA 110 - Project 2
Step 1 - Source
This data is from NYC Open Data, named “Air Quality and Health Impacts”
Direct URL: https://data.cityofnewyork.us/Environment/Air-Quality-and-Health-Impacts/c3uy-2p5r/about_data
The dataset has 12 variables and 19,827 rows containing 1 quantitative variable (Data Value, which holds the value for each indicator) and the rest categorical (Name, Measure, Geo Type Name, Geo Place Name, Time Period).
Variables I plan to use:
- Categorical: Geo Place Name (Community district and Borough)
- Quantitative: Year of the measurement (2009-2024), annual mean fine particles (PM2.5), in µg/m³, annual mean nitrogen dioxide, in ppb, summer mean nitrogen dioxide, in ppb, winter mean nitrogen dioxide, in ppb and summer mean ozone, in ppb.
Step 2 - Image
Step 3 - Topic introduction and dataset
I choose this topic because clean air is directly tied to human health and well being, yet for years, industrial growth prioritized production over environmental pollution. “Annually, ozone pollution in New York City causes around 400 premature deaths, 850 hospitalizations, and around 4,500 emergency visits for health issues such as asthma.” (Ronniya Varghese, 2022)
It is a government job to create strategy to reduce the impact of industries, and for me this theme is meaningful in order to create a healthy and green planet for my future.
| Variable | Meaning | Type |
|---|---|---|
district |
Community district (59 in NYC) | Categorical |
borough |
Borough, created from the first digit of the district code | Categorical |
year |
Year of the measurement (2009-2024) | Quantitative |
pm25 |
Annual mean fine particles (PM2.5), in µg/m³ (response) | Quantitative |
no2 |
Annual mean nitrogen dioxide, in ppb | Quantitative |
no2_summer |
Summer mean nitrogen dioxide, in ppb | Quantitative |
no2_winter |
Winter mean nitrogen dioxide, in ppb | Quantitative |
o3 |
Summer mean ozone, in ppb | Quantitative |
How the data was cleaned? The quantitive variables are written in lines, so I used filter() and select() to create a table for each indicator, then I used left_join() to marge the table with one row per district. After that I used mutate() and case_when() to create year and borough. The analysis table have 8 variables, 2 categorical and 6 quantitative.
Step 4 - Load the libraries and data
# load the dataset
aq <- read_csv("Air_Quality_and_Health_Impacts_20261109.csv")Rows: 19827 Columns: 12
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (7): Name, Measure, Measure Info, Geo Type Name, Geo Place Name, Time Pe...
dbl (3): Unique ID, Geo Join ID, Data Value
num (1): Indicator ID
lgl (1): Message
ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
# checking the data structure
glimpse(aq)Rows: 19,827
Columns: 12
$ `Unique ID` <dbl> 979194, 980097, 980098, 980099, 980100, 980101, 98010…
$ `Indicator ID` <dbl> 1427, 1431, 1431, 1431, 1431, 1431, 1431, 1431, 1431,…
$ Name <chr> "Fine particles (PM 2.5)", "Nitrogen dioxide (NO2)", …
$ Measure <chr> "Winter mean", "Annual mean", "Annual mean", "Annual …
$ `Measure Info` <chr> "mcg/m3", "ppb", "ppb", "ppb", "ppb", "ppb", "ppb", "…
$ `Geo Type Name` <chr> "UHF34", "UHF34", "UHF34", "UHF34", "UHF34", "UHF34",…
$ `Geo Join ID` <dbl> 104, 304, 303, 302, 301, 211, 210, 209, 208, 207, 206…
$ `Geo Place Name` <chr> "Pelham - Throgs Neck", "Upper West Side", "East Harl…
$ `Time Period` <chr> "2024", "2024", "2024", "2024", "2024", "2024", "2024…
$ Start_Date <chr> "01/01/2024", "01/01/2024", "01/01/2024", "01/01/2024…
$ `Data Value` <dbl> 5.834572, 18.288730, 16.184849, 16.521823, 16.264552,…
$ Message <lgl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, N…
There is one numeric column (Data Value), while the indicators are stored in rows
Step 5 - Cleaning the data
# Keep only the community districts
aq1 <- aq |> filter(`Geo Type Name` == "CD")Transforming the rows into tables
# PM2.5 annual mean (the variable I want to predict)
t_pm25 <- aq1 |>
filter(Name == "Fine particles (PM 2.5)", Measure == "Annual mean") |>
select(`Geo Join ID`, `Geo Place Name`, `Time Period`, pm25 = `Data Value`)
# NO2 annual, summer, and winter means
t_no2 <- aq1 |>
filter(Name == "Nitrogen dioxide (NO2)", Measure == "Annual mean") |>
select(`Geo Join ID`, `Time Period`, no2 = `Data Value`)
t_no2_summer <- aq1 |>
filter(Name == "Nitrogen dioxide (NO2)", Measure == "Summer mean") |>
select(`Geo Join ID`, `Time Period`, no2_summer = `Data Value`)
t_no2_winter <- aq1 |>
filter(Name == "Nitrogen dioxide (NO2)", Measure == "Winter mean") |>
select(`Geo Join ID`, `Time Period`, no2_winter = `Data Value`)
# Ozone summer mean
t_o3 <- aq1 |>
filter(Name == "Ozone (O3)", Measure == "Summer mean") |>
select(`Geo Join ID`, `Time Period`, o3 = `Data Value`)creating a new df
# marging the five tables by district and year
air <- t_pm25 |>
left_join(t_no2, by = c("Geo Join ID", "Time Period")) |>
left_join(t_no2_summer, by = c("Geo Join ID", "Time Period")) |>
left_join(t_no2_winter, by = c("Geo Join ID", "Time Period")) |>
left_join(t_o3, by = c("Geo Join ID", "Time Period")) |>
mutate(year = as.numeric(`Time Period`),
#The dataset has no borough column for districts, so I create it from the first digit of the district code
borough = case_when(`Geo Join ID` < 200 ~ "Manhattan",
`Geo Join ID` < 300 ~ "Bronx",
`Geo Join ID` < 400 ~ "Brooklyn",
`Geo Join ID` < 500 ~ "Queens",
TRUE ~ "Staten Island"))|>
rename(district = `Geo Place Name`) |>
select(district, borough, year, pm25, no2, no2_summer, no2_winter, o3) |>
# removing NAs only for the variables I use
filter(!is.na(pm25) & !is.na(no2) & !is.na(no2_summer) &
!is.na(no2_winter) & !is.na(o3))
head(air)# A tibble: 6 × 8
district borough year pm25 no2 no2_summer no2_winter o3
<chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
1 Throgs Neck and Co-op C… Bronx 2024 6.04 14.2 12.2 18.5 34.5
2 Parkchester and Soundvi… Bronx 2024 6.45 14.9 12.8 19.2 33.7
3 Riverdale and Fieldston… Bronx 2024 6.33 13.5 12.2 17.1 32.5
4 Kingsbridge Heights and… Bronx 2024 6.45 14.6 13.2 18.1 33.3
5 Belmont and East Tremon… Bronx 2024 6.63 15.3 13.6 19.1 33.2
6 Fordham and University … Bronx 2024 6.46 16.1 14.6 19.7 32.7
Step 6 - Exploring the data
#how many rows were left
air |> summarise(rows = n())# A tibble: 1 × 1
rows
<int>
1 944
Trend over time
yearly <- air |>
group_by(year) |>
summarise(avg_pm25 = mean(pm25))
ggplot(yearly, aes(x = year, y = avg_pm25)) +
geom_line() +
geom_point() +
labs(title = "Average PM2.5 over time", x = "Year",
y = "Average PM2.5 (µg/m³)") +
theme_minimal(base_size = 12)This plot creates a trend over the years with the average PM2.5. The average for 2009 was 11µg/m³, while in 2024 is approximately 6.5µg/m³. That shows that over the time the levels decrease.
p1 <- ggplot(air, aes(x = no2, y = pm25)) +
geom_point(alpha = 0.4) +
geom_smooth(method = 'lm', formula = y ~ x) + # lm = linear model
labs(title = "PM2.5 VERSUS NO2 IN NYC COMMUNITY DISTRICTS (2009-2024)",
caption = "Source: NYC Open Data, Air Quality and Health Impacts",
x = "Annual mean NO2 (ppb)",
y = "Annual mean PM2.5 (µg/m³)") +
theme_minimal(base_size = 12)
p1The geom plot shows the relationship between annual mean of PM2.5 and NO2. There is a positive trend, meaning that as the NO2 increases, the PM2.5 also tends to increase.
air |>
group_by(borough) |>
summarise(avg_pm25 = mean(pm25)) |>
ggplot(aes(x = borough, y = avg_pm25)) +
geom_bar(stat = "identity", fill = "darkred") +
labs(title = "Average PM2.5 by borough", x = "Borough",
y = "Average PM2.5 (µg/m³)") +
theme_minimal(base_size = 12)The bar plot compares the Average PM2.5 by borough. Manhattan is the biggest bar in this plot, but the overall Average PM2.5 is higher than 7µg/m³.
Multiple linear regression
Checking correlation
cor_matrix <- air |>
select(pm25, no2, no2_summer, no2_winter, o3, year) |>
cor()
round(cor_matrix, 2) pm25 no2 no2_summer no2_winter o3 year
pm25 1.00 0.89 0.87 0.83 -0.51 -0.81
no2 0.89 1.00 0.98 0.94 -0.66 -0.61
no2_summer 0.87 0.98 1.00 0.89 -0.68 -0.55
no2_winter 0.83 0.94 0.89 1.00 -0.68 -0.59
o3 -0.51 -0.66 -0.68 -0.68 1.00 0.27
year -0.81 -0.61 -0.55 -0.59 0.27 1.00
The variable PM2 has a strong correlation between NO2 (0.89), NO2_summer(0.87) and NO2_winter (0.83) with a positive slope. Year (-0.81) also has a strong correlation, but with a negative slope, while O3(-0.51) has a moderate correlation.
The three NO2 predictors have the highest correlation. The collinearity happens because the variables are not independent of each other and explain the same thing for different periods.
Comparing the others predictors: NO2 and year (-0.55 to -0.61) have a negative slope, which happens the same with O3 and NO2 (-0.66 to -0.68) and for O3 and Year there is a little correlation (0.27)
Full model
The model has the equation: PM2.5 = 353.45+ 0.08(NO2)+ 0.15(NO2_SUMMER)+ 0.01(NO2_WINTER)+ 0.03(03)+ (-0.17(YEAR))
# Model 1: all predictors
fit1 <- lm(pm25 ~ no2 + no2_summer + no2_winter + o3 + year, data = air)
summary(fit1)
Call:
lm(formula = pm25 ~ no2 + no2_summer + no2_winter + o3 + year,
data = air)
Residuals:
Min 1Q Median 3Q Max
-1.2451 -0.3820 -0.0440 0.3538 2.2883
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 353.453250 10.608075 33.319 < 2e-16 ***
no2 0.078457 0.036573 2.145 0.0322 *
no2_summer 0.152436 0.025137 6.064 1.92e-09 ***
no2_winter 0.011496 0.014905 0.771 0.4407
o3 0.038277 0.007727 4.954 8.64e-07 ***
year -0.173910 0.005198 -33.454 < 2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.5319 on 938 degrees of freedom
Multiple R-squared: 0.9166, Adjusted R-squared: 0.9162
F-statistic: 2063 on 5 and 938 DF, p-value: < 2.2e-16
The predictor no2_winter had a large p-value, meaning that the predictor does not fit the model.
Removing the predictor with the largest p-value (above 0.05)
The model has the equation: PM2.5 = 352.35+ 0.10(NO2)+ 0.14(NO2_SUMMER)+ 0.04(03)+ (-0.17(YEAR))
fit2 <- lm(pm25 ~ no2 + no2_summer + o3 + year, data = air)
summary(fit2)
Call:
lm(formula = pm25 ~ no2 + no2_summer + o3 + year, data = air)
Residuals:
Min 1Q Median 3Q Max
-1.23423 -0.38135 -0.04404 0.35961 2.29813
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 352.348605 10.508676 33.529 < 2e-16 ***
no2 0.101409 0.021256 4.771 2.13e-06 ***
no2_summer 0.139612 0.018849 7.407 2.88e-13 ***
o3 0.035970 0.007123 5.050 5.32e-07 ***
year -0.173309 0.005139 -33.726 < 2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.5318 on 939 degrees of freedom
Multiple R-squared: 0.9166, Adjusted R-squared: 0.9162
F-statistic: 2579 on 4 and 939 DF, p-value: < 2.2e-16
All the variables have a p-value with three asterisks, with suggests strong evidence that no2, no2_summer, o3 and year are related to pm25. Looking at the adjusted R-squared, 91.6% of the variation in pm25 is explained by the model. In other words, only about 8% of the variation is not explained by this model.
For each additional 1 ppb of NO2, the model predicts an increase of about 0.10 µg/m³ of PM2.5.
Trying a simpler model
The model has the equation: PM2.5 = 328.23+ 0.25(NO2)+ 0.03(03)+ (-0.16(YEAR))
fit3 <- lm(pm25 ~ no2 + o3 + year, data = air)
summary(fit3)
Call:
lm(formula = pm25 ~ no2 + o3 + year, data = air)
Residuals:
Min 1Q Median 3Q Max
-1.29427 -0.41543 -0.03907 0.35477 2.57316
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 328.229879 10.273718 31.949 < 2e-16 ***
no2 0.252769 0.006015 42.024 < 2e-16 ***
o3 0.030950 0.007291 4.245 2.41e-05 ***
year -0.161661 0.005030 -32.137 < 2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 0.5468 on 940 degrees of freedom
Multiple R-squared: 0.9117, Adjusted R-squared: 0.9114
F-statistic: 3235 on 3 and 940 DF, p-value: < 2.2e-16
All the variables still have a p-value with three asterisks, but the adjusted R-squared of 91.1% did not suffer any improvement. We need to check if the simpler model is really the best.
# checking for improvement
anova(fit3, fit2)Analysis of Variance Table
Model 1: pm25 ~ no2 + o3 + year
Model 2: pm25 ~ no2 + no2_summer + o3 + year
Res.Df RSS Df Sum of Sq F Pr(>F)
1 940 281.09
2 939 265.57 1 15.516 54.862 2.876e-13 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The p-value is small, so we choose the fit2. There is strong evidence that Nitrogen dioxide (NO2) measured in summer improves the model.
Step 8 - Final visualization
# Scatterplot with one color for each borough
p <- ggplot(air, aes(x = no2, y = pm25, color = borough,
text = paste("District:", district,
"\nYear:", year,
"\nNO2:", round(no2, 1), "ppb",
"\nPM2.5:", round(pm25, 2), "µg/m³"))) +
geom_point(size = 2, alpha = 0.6) +
scale_color_brewer(name = "Borough", palette = "Set1") +
labs(title = "Districts With Higher NO2 Levels \nAlso Have Higher PM2.5 (2009-2024)",
caption = "Source: NYC Open Data, Air Quality and Health Impacts",
x = "Annual mean nitrogen dioxide, NO2 (ppb)",
y = "Annual mean fine particles, PM2.5 (µg/m³)") +
theme_minimal(base_size = 12)
ggplotly(p, tooltip = "text") # interactive versionWorks cited:
Varghese, Ronniya. “Air Pollution in NYC: Causes, Effects, and Solutions.” Earth.Org, 15 Nov. 2022,https://earth.org/air-pollution-in-nyc/.