hmwork (iii)

Salem AlMheiri

Saeed AlMheiri

Dhabia AlShehhi

Hamdan Kalantar

Introduction

This analysis explores the New York City 311 Service Requests dataset, which contains a wealth of information about non-emergency service requests made by New York City residents. The 311 system, launched in 2003, serves as a crucial link between city residents and government services, handling millions of requests annually across a wide range of issues. Our exploration aims to uncover patterns, trends, and insights within this rich dataset, focusing on several key areas:

  1. Complaint Types: We will investigate the most common types of complaints filed through the 311 system, providing insight into the primary concerns of NYC residents.

  2. Geographic Distribution: By analyzing complaints across boroughs and using geospatial visualization, we’ll explore how issues vary across different parts of the city.

  3. Temporal Patterns: We will examine how complaint volumes and types change over time, looking for daily, seasonal, and long-term trends.

  4. Agency Performance: The analysis will look at which city agencies handle the most complaints and examine response times to different types of issues.

  5. Correlations and Relationships: We will investigate potential relationships between different complaint types, boroughs, and other factors.

This analysis seeks to provide valuable insights for city planners, policymakers, and researchers. By understanding patterns in 311 service requests, we can gain a deeper understanding of urban issues, resource allocation needs, and the evolving concerns of New York City residents.

Install Packages

# Load required packages
library(lubridate)
Warning: package 'lubridate' was built under R version 4.3.3

Attaching package: 'lubridate'
The following objects are masked from 'package:base':

    date, intersect, setdiff, union
library(dplyr)
Warning: package 'dplyr' was built under R version 4.3.3

Attaching package: 'dplyr'
The following objects are masked from 'package:stats':

    filter, lag
The following objects are masked from 'package:base':

    intersect, setdiff, setequal, union
library(ggplot2)
Warning: package 'ggplot2' was built under R version 4.3.3
library(ggmap)
Warning: package 'ggmap' was built under R version 4.3.3
ℹ Google's Terms of Service: <https://mapsplatform.google.com>
  Stadia Maps' Terms of Service: <https://stadiamaps.com/terms-of-service/>
  OpenStreetMap's Tile Usage Policy: <https://operations.osmfoundation.org/policies/tiles/>
ℹ Please cite ggmap if you use it! Use `citation("ggmap")` for details.
library(sf)
Warning: package 'sf' was built under R version 4.3.3
Linking to GEOS 3.11.2, GDAL 3.8.2, PROJ 9.3.1; sf_use_s2() is TRUE
library(viridis)
Warning: package 'viridis' was built under R version 4.3.3
Loading required package: viridisLite
library(corrplot)
Warning: package 'corrplot' was built under R version 4.3.3
corrplot 0.92 loaded
library(scales)
Warning: package 'scales' was built under R version 4.3.3

Attaching package: 'scales'
The following object is masked from 'package:viridis':

    viridis_pal

Initialization and Data Loading

# Read the full dataset
nyc311<- read.csv("~/nyc311.data.csv")

# Fix column names
names(nyc311) <- names(nyc311) |>
  stringr::str_replace_all("\\s", "")

Data Overview

Sample of the Data

Created.Date Closed.Date Agency Agency.Name Complaint.Type Descriptor Incident.Zip Status Borough Latitude Longitude
04/14/2015 02:14:40 AM 04/14/2015 03:03:22 AM NYPD New York City Police Department Vending In Prohibited Area 10465 Closed BRONX 40.82573 -73.82111
04/14/2015 02:10:12 AM NYPD New York City Police Department Blocked Driveway No Access 11234 Open BROOKLYN 40.61879 -73.93771
04/14/2015 02:03:01 AM NYPD New York City Police Department Noise - Street/Sidewalk Loud Music/Party 11204 Open BROOKLYN 40.61859 -73.99846
04/14/2015 02:02:40 AM NYPD New York City Police Department Noise - Street/Sidewalk Loud Talking 11211 Assigned BROOKLYN 40.71410 -73.95589
04/14/2015 02:00:04 AM 04/14/2015 02:47:33 AM NYPD New York City Police Department Noise - Street/Sidewalk Loud Talking 10025 Closed MANHATTAN 40.79792 -73.96385
04/14/2015 01:52:15 AM 04/14/2015 02:11:10 AM NYPD New York City Police Department Noise - Street/Sidewalk Loud Talking 11205 Closed BROOKLYN 40.68833 -73.96481

Data Dictionary

The dataset captures 311 complaints in New York City, with each row representing a unique incident. Key columns include Created.Date (when the complaint was filed), Closed.Date (when it was resolved), Agency and Agency.Name (handling department), Complaint.Type and Descriptor (nature of the issue), Incident.Zip and Borough (location), Status (Open/Closed/Assigned), and Latitude/Longitude (precise coordinates).

Exploratory Data Analysis

1. Most Common Complaints

nyc311 |>
  count(Complaint.Type, sort = TRUE) |>
  top_n(10) |>
  ggplot(aes(x = reorder(Complaint.Type, n), y = n)) +
  geom_col(fill = "skyblue") +
  coord_flip() +
  labs(title = "Top 10 Most Common Complaint Types",
       x = "Complaint Type",
       y = "Number of Complaints") +
  theme_minimal()
Selecting by n

The bar chart displays the top 10 most common complaint types received by NYC’s 311 service. HEAT/HOT WATER issues are by far the most frequently reported problem, with more than double the number of complaints compared to the next most common issue, Street Condition, indicating that heating and hot water problems are a significant concern for New York City residents.

2. Complaints by Borough

nyc311 |>
  count(Borough) |>
  ggplot(aes(x = reorder(Borough, n), y = n)) +
  geom_col(fill = "lightgreen") +
  coord_flip() +
  labs(title = "Number of Complaints by Borough",
       x = "Borough",
       y = "Number of Complaints") +
  theme_minimal()

The bar chart illustrates the distribution of 311 complaints across New York City’s boroughs, with Brooklyn receiving the highest number of complaints, followed by Queens and the Bronx. Interestingly, Manhattan has fewer complaints than the Bronx despite its dense population, while Staten Island has the lowest number of complaints, which could be due to its smaller population or other factors affecting complaint reporting rates.

3. Time Series Analysis of Complaints

# Convert Created.Date to Date type
dates <- (nyc311$Created.Date)

# Count occurrences of each date
date_counts <- table(dates)

# Create a data frame for plotting
plot_data <- data.frame(
  Date = as.Date(names(date_counts)),
  Count = as.numeric(date_counts)
)

# Sort the data frame by date
plot_data <- plot_data[order(plot_data$Date), ]

# Create the plot
library(ggplot2)

ggplot(plot_data, aes(x = Date, y = Count)) +
  geom_line(color = "blue") +
  labs(title = "Number of Complaints Over Time",
       x = "Date",
       y = "Number of Complaints") +
  theme_minimal()
Warning: Removed 198807 rows containing missing values or values outside the scale range
(`geom_line()`).

The time series graph shows the number of 311 complaints in New York City over time, with a clear pattern of high variability and frequent spikes in complaint numbers. There appears to be a significant drop in complaint volume around 2008-2009, followed by a period of relatively low and stable numbers, before a sharp increase at the very end of the time series, which could indicate a recent surge in complaint submissions or possibly a data anomaly that requires further investigation.

4. Complaints by Agency

nyc311 |>
  count(Agency.Name, sort = TRUE) |>
  top_n(10) |>
  ggplot(aes(x = reorder(Agency.Name, n), y = n)) +
  geom_col(fill = "orange") +
  coord_flip() +
  labs(title = "Top 10 Agencies by Number of Complaints",
       x = "Agency",
       y = "Number of Complaints") +
  theme_minimal()
Selecting by n

The bar chart shows that the Department of Housing Preservation and Development receives significantly more complaints than any other agency in New York City, with more than double the number of complaints compared to the second-ranked Department of Transportation. This suggests that housing-related issues are the most prevalent concerns among NYC residents using the 311 service, potentially indicating widespread problems with housing conditions or management in the city.

5. Complaint Types by Borough

nyc311 |>
  group_by(Borough, Complaint.Type) |>
  summarise(count = n(), .groups = 'drop') |>
  group_by(Borough) |>
  top_n(5, count) |>
  ggplot(aes(x = reorder(Complaint.Type, count), y = count, fill = Borough)) +
  geom_col() +
  facet_wrap(~ Borough, scales = "free_y") +
  coord_flip() +
  labs(title = "Top 5 Complaint Types by Borough",
       x = "Complaint Type",
       y = "Number of Complaints") +
  theme_minimal() +
  theme(legend.position = "none")

Bronx: The top complaint in the Bronx is HEAT/HOT WATER, followed by STREET CONDITION. UNSANITARY CONDITION, PLUMBING, and PAINT/PLASTER round out the top 5 complaints in this borough.

Brooklyn: Similar to the Bronx, HEAT/HOT WATER is the most common complaint in Brooklyn. However, the second most frequent complaint is UNSANITARY CONDITION, followed by STREET CONDITION, STREET LIGHT CONDITION, and PAINT/PLASTER.

Manhattan: Manhattan shows a slightly different pattern. While HEAT/HOT WATER remains the top complaint, PAINT/PLASTER is the second most common issue. STREET CONDITION, NOISE - COMMERCIAL, and NOISE follow as the next most frequent complaints.

Queens: In Queens, HEAT/HOT WATER is again the primary complaint. STREET CONDITION is second, followed by STREET LIGHT CONDITION, ILLEGAL PARKING, and WATER SYSTEM.

Staten Island: Staten Island shows a unique pattern compared to the other boroughs. STREET CONDITION is the top complaint, followed by STREET LIGHT CONDITION. HEAT/HOT WATER is only the third most common complaint here, with ILLEGAL PARKING and NOISE following.

Unspecified: For complaints where the borough is unspecified, DOF Literature Request is the most common, followed by BENEFIT CARD REPLACEMENT. Other top issues include GENERAL CONSTRUCTION, AGENCY ISSUES, and HPD Literature Request.

6. Response Time Analysis

nyc311 |>
  mutate(
    Created.Date = as.POSIXct(Created.Date, format = "%m/%d/%Y %I:%M:%S %p"),
    Closed.Date = as.POSIXct(Closed.Date, format = "%m/%d/%Y %I:%M:%S %p"),
    Response.Time = as.numeric(difftime(Closed.Date, Created.Date, units = "hours"))
  ) |>
  filter(!is.na(Response.Time) & Response.Time >= 0 & Response.Time <= 720) |> # Filter out unreasonable values
  ggplot(aes(x = Response.Time)) +
  geom_histogram(binwidth = 24, fill = "purple", alpha = 0.7) +
  labs(title = "Distribution of Response Times",
       x = "Response Time (hours)",
       y = "Count") +
  theme_minimal()

The histogram shows the distribution of response times for some type of service or complaint system, measured in hours. The data reveals that the vast majority of responses occur very quickly, within the first few hours, with a sharp decline in frequency as response time increases, although there are some cases with much longer response times extending out to around 600 hours.

7. Geospatial Analysis

library(ggplot2)
library(maps)
Warning: package 'maps' was built under R version 4.3.3

Attaching package: 'maps'
The following object is masked from 'package:viridis':

    unemp
library(dplyr)

# Select zip codes and complaint types
compByZip <- nyc311 |> 
  select(Incident.Zip, Complaint.Type) |> 
  group_by(Incident.Zip) |> 
  summarize(count = n())

# Read the shapefile
spdf <- c("AIzaSyAUMGvS8amdDROfhFbBIwnXGImxkksxE0U")
# Sample the data for one type of complaint
sampled_data <- nyc311 |>
  filter(Complaint.Type == "Noise - Residential") |>
  sample_n(min(1000, n()))

# Get New York map data
ny_map <- map_data("state", region = "new york")

# Create the plot
ggplot() +
  geom_polygon(data = ny_map, aes(x = long, y = lat, group = group), 
               fill = "red", color = "blue") +
  geom_point(data = sampled_data, aes(x = Longitude, y = Latitude), 
             size = 2, alpha = 0.5, color = "red") +
  coord_fixed(1.3, xlim = c(-74.3, -73.7), ylim = c(40.5, 40.9)) +
  labs(title = "Map of Residential Noise Complaints in NYC",
       x = "Longitude", y = "Latitude") +
  theme_minimal() +
  theme(panel.background = element_rect(fill = "aliceblue"),
        panel.grid = element_blank())

The map reveals that residential noise complaints in New York City are not evenly distributed, with significant clustering in Manhattan and parts of Brooklyn. This concentration likely reflects the higher population density and mixed-use urban environments in these areas, while the outer boroughs and less densely populated regions show fewer reported incidents.

ggplot(nyc311, aes(x = X.Coordinate..State.Plane., y = Y.Coordinate..State.Plane.)) +
  geom_point(color = "darkred", alpha = 0.5) +
  labs(title = "Spatial Distribution of Complaints",
       x = "Longitudes",
       y = "Latitudes") +
  theme_minimal()
Warning: Removed 105360 rows containing missing values or values outside the scale range
(`geom_point()`).

Advanced Analysis

8. Correlation Between Complaint Types and Boroughs

# Load required packages 
library(tidyverse)
Warning: package 'tidyverse' was built under R version 4.3.3
Warning: package 'tidyr' was built under R version 4.3.3
Warning: package 'readr' was built under R version 4.3.3
Warning: package 'stringr' was built under R version 4.3.3
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ forcats 1.0.0     ✔ stringr 1.5.1
✔ purrr   1.0.2     ✔ tibble  3.2.1
✔ readr   2.1.5     ✔ tidyr   1.3.1
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ readr::col_factor() masks scales::col_factor()
✖ purrr::discard()    masks scales::discard()
✖ dplyr::filter()     masks stats::filter()
✖ dplyr::lag()        masks stats::lag()
✖ purrr::map()        masks maps::map()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(corrplot)
library(tibble)

complaint_borough_matrix <- nyc311 |>
  group_by(Complaint.Type, Borough) |>
  summarise(count = n(), .groups = 'drop') |>
  pivot_wider(names_from = Borough, values_from = count, values_fill = 0) |>
  tibble::column_to_rownames("Complaint.Type")

cor_matrix <- cor(complaint_borough_matrix)

corrplot(cor_matrix, method = "color", type = "upper", order = "hclust",
         tl.col = "black", tl.srt = 45, addCoef.col = "black")

This correlation matrix visualizes the relationships between complaint patterns across different boroughs of New York City. Manhattan, Bronx, and Brooklyn show strong positive correlations (0.95-0.96) in their complaint patterns, indicating similar types and frequencies of issues reported. Queens and Staten Island have weaker correlations with the other boroughs, suggesting more distinct complaint patterns, with Staten Island showing the lowest correlations overall.

9. Seasonal Patterns in Complaints

nyc311 |>
  mutate(
    Created.Date = mdy_hms(Created.Date),
    Month = factor(month(Created.Date), levels = 1:12, labels = month.abb),
    Year = year(Created.Date)
  ) |>
  group_by(Year, Month) |>
  summarise(Complaints = n(), .groups = 'drop') |>
  ggplot(aes(x = Month, y = Complaints, group = Year, color = as.factor(Year))) +
  geom_line() +
  labs(title = "Seasonal Patterns in Complaints",
       x = "Month", y = "Number of Complaints",
       color = "Year") +
  theme_minimal()

The graph reveals a clear seasonal pattern in 311 complaints, with peaks during summer months (July-August) and troughs in winter (December-January), consistent across both years shown. This pattern likely reflects the impact of weather on urban activities and issues, with warmer months generating more complaints possibly due to increased outdoor activities and heat-related problems, while colder months see a reduction, potentially due to less outdoor activity and holiday periods.

Conclusion

This analysis of New York City’s 311 service request data has revealed significant insights into urban issues and resident concerns. The prevalence of heating and hot water complaints, particularly in boroughs like Brooklyn and the Bronx, highlights critical infrastructure challenges faced by many New Yorkers. Clear seasonal patterns in complaint volumes, with peaks in summer and troughs in winter, demonstrate how weather and urban activities influence citizen reporting behaviors. These findings, along with the variations in complaint types across boroughs and the responsiveness of city agencies, provide valuable information for policymakers and urban planners to better allocate resources and address the most pressing concerns of New York City residents.