Class 607, Project 2 Data Transformations

Author

Troy Tournat

Published

November 10, 2026

Instructions here: https://brightspace.cuny.edu/d2l/le/content/1404182/viewContent/41373795/View


Introduction

Data sources: All datasets originated from students in Discussion #5a.

Dataset 1 from student Patricio Romero: “For this discussion, I selected the Electric Consumption and Cost dataset from NYC Open Data. This dataset contains electrical energy information from New York City Housing Authority (NYCHA) buildings.

Analysis request: compare electricity consumption between NYC boroughs and see how electricity use changes over time.

Data Source: NYC Open Data – Electric Consumption And Cost (2010–Sep 2025)

Dataset Link: https://data.cityofnewyork.us/Housing-Development/Electric-Consumption-And-Cost-2010-Sep-2025-/jr24-e7cr

Dataset 2 from student Renee Watson: “fictional dataset containing monthly patient visits for community health clinics during 2025

Analysis request: I would examine how patient visit volume changes throughout the year and compare these patterns across the five clinics. I could determine which clinics experienced the largest increases or decreases in patient visits and examine whether there are noticeable monthly patterns in patient volume. A visualization could also be used to compare trends across the clinics.”  

Dataset 3 from student Kingsley Kpoku: “Fictional NYC housing prices dataset for the five New York Boroughs .The dataset is on 2024 and 2025.

Analysis request: The whole idea is to analyze and study the behavioral patterns of the averages prices across the months and Years and serve as a real estate analyst to advice potential customers on the most expensive areas in NewYork, what people can afford based on salary and when is the best time to move to a borough when the prices drop (based on historicals)”

*Dataset 4 from student Noelle Messaoudi: “As a single mom who works part-time and attends graduate school part-time, I often have to fit my workload and studying around my toddler’s sleep. This gave me the idea to create a fictional dataset based on my experience of trying to manage everything in one day.

The dataset would record a child’s sleep over two weeks. It would be untidy because each day’s nap and bedtime would have separate columns, instead of having one row for each day.

Analysis request: For Project 2, I could explore whether longer naps are associated with later bedtimes and how much the sleep schedule varies from day to day.I am curious about this because “I will study when he sleeps” sounds like a plan, until he decides otherwise.”

*adding one more since Dataset 1 wasn’t a wide dataset. Since this data is an example, I used Claude to expand on it.

Tackling the problem: In all three datasets I’ll need to extract the wide, messy data in different ways from online (dataset #1), the discussion body (dastaset #2), or downloaded (dataset #3). Then, I’ll import them all in R and begin reshaping and I’ll review the different analysis requests to determine what more is needed for preparation. Finally, I’ll tackle the analysis questions with a summary table, visualization, and short narrative.

Anticipated data challenges: Dataset1 seems like it will be the most challenging because I don’t know anything about how to analyize energy consumption, and the timespan is 15 years, so it requires deduplicating and sifting through a lot of data. I also anticipate all the datacleaning across the three datasets to be challenging.

Citations:

Analysis

Load Packages

#Install packages needed 
pacman::p_load(
  # Package Install and Management
  pacman,           # package install/load
  janitor,          #clean up data names 
  
  # Project and File Management
  readr,            # import data
  httr,             #github passkey 
  usethis,          #r environment
  RPostgres,        #SQL 
  DBI,              #SQL 
  jsonlite,         #working with json
  rvest,           #html files
  rstuioapi,        #api
  knitr,            #displaying data in quarto
  
  # General Data Management
  tidyverse,        # work with dates
  scales,           # ggplot 
  gmodels,          # freq tables SAS 
  skimr, 
  hms,              # recommended for dataset 4 
  patchwork        #data visualization, side by side 

)

Load Data

First, I am deduplicating and dropping NAs to dataset1 so it can be uploaded to github.

#locations 
#input_location <- "~/Downloads/"
#location_desktop <- "~/Desktop/"


#dataset1 <- readr::read_csv(paste0(location_desktop, "dataset1.csv"), show_col_types = FALSE)

#look at date variable 
#skim(dataset1$`Revenue Month`)

#prepare for creating a subset with unique bill ids and no na 
#dataset1_sub <- dataset1 %>%
#  clean_names() %>%
#  distinct(umis_bill_id, .keep_all = TRUE)%>% 
#  drop_na()

  
#check date range 
#range(dataset1_sub$revenue_month)

#create subset file 
#write_csv(dataset1_sub, file.path(location_desktop, paste0("dataset1_sub.csv")))

Next, load data saved as .csv in github and import into R

location <- "~/Desktop/CUNY/607-assignments/projects/project 2/"


#Dataset 1: 
dataset1 <- read_csv(paste0(location, "dataset1_sub.csv"), show_col_types = FALSE)
#Dataset 2:
dataset2 <- read_csv(paste0(location, "dataset2.csv"), show_col_types = FALSE)
#Dataset 3:
dataset3 <- read_csv(paste0(location, "dataset3.csv"), show_col_types = FALSE)
#Dataset 4: 
dataset4 <- read_csv(paste0(location, "dataset4.csv"), show_col_types = FALSE)

#

#

Dataset 1

Data cleaning

#create additional variables that we need for the analysis. 
dataset1_sub <- dataset1 %>%
  mutate(
    revenue_month = ym(revenue_month),
    building_id = paste(coalesce(as.character(tds_number), as.character(edp)),
                        location, sep = "-"),
    # use per day, since bills cover different numbers of days
    consumption_per_day = consumption_kwh / number_days)
    
    

#take care of invalid, NAs, duplicates (see checks)
#Keep the highest bill ID per building, meter and revenue month
dataset1_2sub <- dataset1_sub%>% 
  group_by(building_id, meter_number, revenue_month) %>%
  slice_max(umis_bill_id, n = 1, with_ties = FALSE) %>% 
  #drop nas and any 0 values 
  drop_na(borough, revenue_month, consumption_kwh, number_days) %>%
  filter(consumption_kwh > 0, number_days > 0) %>%
  ungroup() 



#now create dataset and add variables 
dataset1_3sub <- dataset1_2sub |>
  #add number of buildings per borough and total consumption  
  group_by(borough, revenue_month) %>%
  summarize(
    n_buildings    = n_distinct(building_id),
    mo_total_kwh   = sum(consumption_kwh),
    mo_kwh_per_day = sum(consumption_per_day),
    .groups = "drop"
  ) %>% 
  mutate(
    kwh_per_bldg_per_day = mo_kwh_per_day / n_buildings, 
    # date and season
    rev_year   = year(revenue_month),
    rev_month  = month(revenue_month),
    rev_season = case_when(rev_month %in% c(12, 1, 2) ~ "winter",
                           rev_month %in% 3:5  ~ "spring",
                           rev_month %in% 6:8  ~ "summer",
                           rev_month %in% 9:11 ~ "fall"), 
    # year-over-year change on that measure
    kwh_last_year  = lag(kwh_per_bldg_per_day, 12),
    yoy_pct_change = (kwh_per_bldg_per_day - kwh_last_year) / kwh_last_year * 100)%>%
  ungroup()


#CHECKS FOR DUPLICATES
dataset1_2sub %>% count(umis_bill_id) %>% count(n, name = "n_bill_ids")

#checking building rows per month 
dataset1_2sub %>%
  count(building_id, revenue_month) %>%
  count(n, name = "n_building_months")

#should be none
dup_check <- dataset1_2sub %>%
  group_by(building_id, meter_number, revenue_month) %>%
  filter(n() > 1) %>%
  arrange(building_id, meter_number, revenue_month)

Run analysis

Compare electricity consumption between NYC boroughs and see how electricity use changes over time.

Table:
annual_long <- dataset1_3sub %>%
  # 2025 only has data up to Sept so exclude. Also exclude non development city as a borough 
  filter(rev_year < 2025, borough != "NON DEVELOPMENT FACILITY") %>%               
  group_by(borough, rev_year) %>%
  summarize(avg_kwh = mean(kwh_per_bldg_per_day, na.rm = TRUE), .groups = "drop")


#create table
annual_wide <- annual_long %>%
  select(rev_year, borough, avg_kwh)%>% 
  pivot_wider(names_from = c(borough), values_from = avg_kwh)

# display in quarto
kable(annual_wide, 
      col.names = c("Year", "Bronx", "Brooklyn", "Manhattan", "Queens", "Staten Island"),
      caption = "Annual Average Daily Electricity Use (kWh) per Building, 2010 - 2024") 
Annual Average Daily Electricity Use (kWh) per Building, 2010 - 2024
Year Bronx Brooklyn Manhattan Queens Staten Island
2010 1455.492 186.9330 462.2863 2822.082 636.0199
2012 1360.134 191.0044 506.3698 2603.030 714.3561
2013 1294.160 192.3820 531.2423 2469.217 605.7245
2014 1220.432 198.1961 517.3521 2461.666 573.8604
2015 1221.958 194.1154 1117.9037 2132.105 1384.7514
2016 1189.571 183.7798 1057.4120 2378.037 1280.1470
2017 1142.234 180.3669 1011.2828 2397.003 1298.2257
2019 1111.510 176.3655 983.2447 2239.892 1129.9895
2020 1185.011 162.6253 944.8685 1912.892 1295.4225
2021 1188.835 160.9826 933.4759 1857.763 1183.4413
2022 1179.243 176.1399 895.3775 1877.424 1157.8735
2023 1619.539 1159.6294 365.1579 1065.870 1244.7210
2024 1061.356 168.8994 839.2531 1465.844 1208.2962
Graph:
ggplot(annual_long, aes(rev_year, avg_kwh, color = borough)) +
  geom_line(linewidth = 0.9) + geom_point() +
  scale_y_continuous(labels = comma) +
  labs(title = "Annual Average Daily Electricity Use per Building 2010 - 2024",
       x = "Year", y = "kWh per building per day", color = "Borough") +
  theme_minimal()

Summary

Overall we see that yearly average energy consumption (KWh per building/day) is different by borough. Notably Queens has steadily decreased from approximately 2750 in 2010 to 1500 in 2024. in Brooklyn, energy consumption has stayed low until 2023 and then spiked up above 1000 in 2023.

#

#

Dataset 2

Data cleaning

#clean headers 
dataset2_sub <- dataset2 %>% clean_names()

  
#transform the dataset from wide to long to see all dates in one column 
dataset2_long <- dataset2_sub %>%
  pivot_longer(
    cols = !clinic,
    names_to = "month",
    values_to = "count")%>% 
  #edit month to date variables and add percents 
 mutate(
   #month_num = match(str_to_title(str_sub(month, 1, 3)), month.abb), 
   month = factor(str_to_title(str_sub(month, 1, 3)), levels = month.abb), 
   totals = sum(count), .by= clinic, 
   pct_total = round((count / totals) *100, 1))

Run analysis

How do patient visit volume changes throughout the year? Compare these patterns across the five clinics.

Table:
#display work with  this 
dataset2_long|> 
  select(clinic, month, pct_total)%>% 
  kable(col.names = c("clinic", "month", "pct_total")) 


#create table
dataset2_table <- dataset2_long %>%
  select(clinic, month, pct_total)%>% 
  pivot_wider(names_from = c(clinic), values_from = pct_total)
# display in quarto
kable(dataset2_table, 
      caption = "Monthly Percent of Patient Visit Volume, by Clinic") 
Monthly Percent of Patient Visit Volume, by Clinic
month Harbor Community Health Center Riverside Family Health Clinic Central Community Health Center Eastside Family Clinic Westview Community Health Center Northside Health Clinic Greenwood Family Health Center Lakeside Community Clinic Parkview Health Center Hillcrest Community Clinic
Jan 7.1 6.6 7.2 6.7 7.1 7.0 6.7 7.2 6.9 6.6
Feb 6.7 7.1 6.9 7.0 6.9 6.8 6.6 7.0 7.1 6.8
Mar 7.5 6.9 7.6 7.3 7.3 7.4 7.2 7.4 7.4 7.3
Apr 8.1 7.7 7.9 7.2 7.7 7.8 7.7 7.8 7.7 7.6
May 8.4 8.2 8.3 7.8 8.1 8.1 8.1 8.0 8.1 7.9
Jun 8.7 8.4 8.1 8.4 8.4 8.4 8.2 8.4 8.4 8.2
Jul 9.1 8.7 8.6 8.7 8.3 8.7 8.6 8.6 8.2 8.6
Aug 9.0 9.1 8.5 9.1 8.7 8.5 8.9 8.5 8.7 8.8
Sep 8.3 8.9 8.8 8.8 9.0 9.0 9.1 8.9 9.0 9.1
Oct 8.6 9.4 9.1 9.4 9.2 9.2 9.4 9.2 9.2 9.4
Nov 9.0 9.3 9.3 9.7 9.4 9.4 9.6 9.4 9.6 9.6
Dec 9.4 9.7 9.7 9.9 9.8 9.8 9.9 9.7 9.8 10.0
Graph:
ggplot(dataset2_long, aes(month, pct_total, color = clinic, group = clinic)) +
  geom_line(linewidth = 0.9) + geom_point() +
  labs(title = "Monthly Patient Visit Volume by Clinic in 2025",
       x = "Month", y = "Percent of clinic's yearly total", color = "Clinic") +
  theme_minimal()

Summary

At all clinics, patient visits saw an upward trend from 7% to 10% by the end of the year, with a slight dip in February. Harbor Community Health Center saw the highest volume of patient visits from March to July then switched to the lowest visit percentage from September to December.

#

#

Dataset 3

Data cleaning

#clean headers 
dataset3_sub <- dataset3 %>% clean_names()

#weird formatting split the dataset into two parts to fix 
first_part  <- dataset3_sub %>% slice(1:24)
second_part <- dataset3_sub %>% slice(25:n())

#fix second part 
second_part_wide <- second_part%>% 
  select(year, month, bronx, brooklyn) %>%
  pivot_wider(
     names_from = bronx, 
     values_from = brooklyn)

#reclean headers 
second_part_wide <- second_part_wide%>% clean_names()

#change bronx to numeric in both parts 
first_part$bronx <- as.numeric(first_part$bronx)
second_part_wide$bronx <- as.numeric(second_part_wide$bronx)


#put back together with first part 
dataset3_clean <- bind_rows(first_part, second_part_wide)

#edit and add variables  
dataset3_clean <- dataset3_clean %>%
  mutate( 
    month = factor(str_to_title(str_sub(month, 1, 3)), levels = month.abb), 
    date = make_date(year, month, 1), 
    year = format(date, "%Y"),
    date_label = format(date, "%Y-%m"))%>% 
  select(-c(month, date))

#pivot longer 
dataset3_long <- dataset3_clean %>%
  pivot_longer(
    cols = -c(date_label,year),
    names_to = "borough",
    values_to = "price")%>% 
  # keep one row per borough-month 
  arrange(borough, date_label, is.na(price)) %>%
  distinct(borough, date_label, .keep_all = TRUE) %>%
  #year total by borough 
  mutate(
    yr_totals    = sum(price, na.rm = TRUE),
    yr_pct_total = round(price / yr_totals * 100, 1),
    .by = c(borough, year)
  ) %>%
  # month total
  mutate(
    mo_totals    = sum(price, na.rm = TRUE),
    mo_pct_total = round(price / mo_totals * 100, 1),
    .by = date_label
  )%>% 
  #make date_label back to date 
  mutate(
    date_label = ym(date_label)
  )

Run analysis

What are the average prices across the months and years in each NYC borough? Is there a best time to move to a borough when the prices drop?

Table:
#create table
dataset3_sum <- dataset3_long %>%
  summarise(
    avg_price = round(mean(price), 1), 
    #avg_mo_pct = round(mean(mo_pct_total),1), 
    n = sum(!is.na(price)), 
    .by = c(year, borough))%>% 
  arrange(year, borough)%>% 
  select(-n)
    

# display in quarto
kable(dataset3_sum, 
      col.names = c("Year", "Borough", "Avg Price"),
      caption = "Yearly Average Real Estate Prices, by Borough") 
Yearly Average Real Estate Prices, by Borough
Year Borough Avg Price
2024 bronx 580.6
2024 brooklyn 975.0
2024 manhattan 1130.4
2024 queens 669.7
2024 staten_island 695.6
2025 bronx 608.7
2025 brooklyn 1025.2
2025 manhattan 1174.8
2025 queens 704.0
2025 staten_island 733.4
Graph:
ggplot(dataset3_long, aes(date_label, mo_pct_total, color = borough, group = borough)) +
  geom_line(linewidth = 0.9) + geom_point() +
  scale_x_date(date_breaks = "3 months", date_label = "%Y-%m") +
  labs(title = "Monthly Percent of Average Real Estate Prices, by Borough",
       x = "Year-Month", y = "Percent of Average Real Estate Prices", color = "Borough") +
  theme_minimal()

Summary

The average prices across each NYC borough from 2024-25 stayed relatively the same across time, but there were differences by borough. The most expensive average price was manhattan, followed by brooklyn. The lowest average place was in the bronx.

#

#

Dataset 4

Data cleaning

#clean headers 
dataset4_sub <- dataset4 %>% clean_names()

#transform the dataset from wide to long to see all dates in one column 
dataset4_long <- dataset4_sub %>%
  pivot_longer(
    cols = matches("^week_\\d+$"), 
    names_to = "week",
    names_transform = readr::parse_number,
    values_to = "time", 
    values_transform = as.character)%>% 
  #day_measure combines both measures, separate 
  separate(day_measure, into = c("day", "measure"), sep = " ", extra = "merge")%>%
  pivot_wider(names_from = measure, 
              values_from = time)%>% 
  clean_names()



#add and edit time variables, determine wake time and sleep time. Package recommendation was hms for time mutations
dataset4_long <- dataset4_long %>%
  mutate(
    # "No nap" becomes NA
    nap_start = na_if(nap_start, "No nap"),
    nap_min   = as.numeric(nap_min),
    nap_hrs = round(nap_min / 60, 2), 
    day       = factor(day, levels = c("Monday", "Tuesday", "Wednesday", "Thursday",
                                       "Friday", "Saturday", "Sunday")),
    # actual time-of-day values (hms class, prints as 12:45:00 / 19:30:00)
    nap_start_time = as_hms(parse_date_time(nap_start, "I:M p")),
    bedtime_time   = as_hms(parse_date_time(bedtime,   "I:M p")),
    # 24-hour text labels, e.g. "12:45" and "19:30"
    nap_start_24 = format(parse_date_time(nap_start, "I:M p"), "%H:%M"),
    bedtime_24   = format(parse_date_time(bedtime,   "I:M p"), "%H:%M"),
    # nap end time = start + length (nap_min)
    nap_end_time = as_hms(as.numeric(nap_start_time) + nap_min * 60), 
    awake_after_nap_min = as.numeric(bedtime_time - nap_end_time) / 60,
    awake_after_nap_hr  = round(awake_after_nap_min / 60, 2)
  ) 

Run analysis

Are longer naps are associated with later bedtimes/staying up longer?

Table:
dataset4_year_average <- dataset4_long %>%
  group_by() %>%
  summarize(
    avg_nap_hrs = round(mean(nap_hrs, na.rm = TRUE), 2),
    avg_awake_hrs = round(mean(awake_after_nap_hr, na.rm = TRUE), 2),
    .groups = "drop"
  )%>% 
  select(avg_nap_hrs, avg_awake_hrs )
dataset4_year_average
#center around average year nap hrs and awake hours 


#calculate averages 
dataset4_average <- dataset4_long %>%
  group_by(week) %>%
  summarize(
    avg_nap_hrs = round(mean(nap_hrs, na.rm = TRUE), 2),
    avg_awake_hrs = round(mean(awake_after_nap_hr, na.rm = TRUE), 2),
    .groups = "drop"
  )%>% 
  mutate(
    nap_centered   = round((avg_nap_hrs   - 1.73) / sd(avg_nap_hrs),   2),
    awake_centered = round((avg_awake_hrs - 4.82) / sd(avg_awake_hrs), 2)   
  )%>%
  select(week, avg_nap_hrs, avg_awake_hrs, nap_centered, awake_centered )


kable(dataset4_average,
      col.names = c("Week (in calendar year)", "Avg nap (hrs)", "Avg awake after nap (hrs)", "Avg nap centered", "Avg awake after nap centered"), 
      caption = "Average nap length and awake time before bed, by week")
Average nap length and awake time before bed, by week
Week (in calendar year) Avg nap (hrs) Avg awake after nap (hrs) Avg nap centered Avg awake after nap centered
1 2.18 4.61 1.82 -1.18
2 1.93 4.46 0.81 -2.03
3 2.11 4.79 1.54 -0.17
4 2.04 4.64 1.25 -1.01
5 1.93 4.64 0.81 -1.01
6 2.11 4.64 1.54 -1.01
7 1.93 4.86 0.81 0.23
8 2.04 4.79 1.25 -0.17
9 2.07 4.54 1.37 -1.58
10 1.61 4.88 -0.48 0.34
11 2.00 4.79 1.09 -0.17
12 2.18 4.50 1.82 -1.80
13 2.00 4.57 1.09 -1.41
14 1.96 4.71 0.93 -0.62
15 1.82 4.89 0.36 0.39
16 1.75 4.83 0.08 0.06
17 1.64 4.75 -0.36 -0.39
18 1.57 5.00 -0.65 1.01
19 1.54 5.08 -0.77 1.46
20 1.75 4.58 0.08 -1.35
21 1.86 4.75 0.53 -0.39
22 1.54 5.04 -0.77 1.24
23 1.71 4.86 -0.08 0.23
24 1.71 5.00 -0.08 1.01
25 1.93 4.82 0.81 0.00
26 1.79 4.75 0.24 -0.39
27 1.86 4.86 0.53 0.23
28 1.57 5.00 -0.65 1.01
29 1.39 4.88 -1.37 0.34
30 1.61 4.75 -0.48 -0.39
31 1.86 4.71 0.53 -0.62
32 1.36 4.65 -1.50 -0.96
33 1.64 4.62 -0.36 -1.13
34 1.86 4.50 0.53 -1.80
35 1.68 5.04 -0.20 1.24
36 1.89 4.89 0.65 0.39
37 1.46 4.83 -1.09 0.06
38 1.36 4.65 -1.50 -0.96
39 1.43 5.04 -1.21 1.24
40 1.64 4.75 -0.36 -0.39
41 1.43 4.96 -1.21 0.79
42 1.61 5.21 -0.48 2.20
43 1.14 4.80 -2.38 -0.11
44 1.79 4.79 0.24 -0.17
45 1.82 4.89 0.36 0.39
46 2.00 4.82 1.09 0.00
47 1.54 4.92 -0.77 0.56
48 1.68 5.11 -0.20 1.63
49 1.36 4.92 -1.50 0.56
50 1.64 5.04 -0.36 1.24
51 1.39 5.00 -1.37 1.01
52 1.43 5.12 -1.21 1.69
Graph:
#not centered 

dataset4_average %>%
  ggplot(aes(week, avg_nap_hrs)) +
  geom_line() + geom_point() +
  labs(x = "Week", y = "Avg nap time (hrs)",
       title = "Average Nap time, by week")

dataset4_average %>%
  ggplot(aes(week, avg_awake_hrs)) +
  geom_line() + geom_point() +
  labs(x = "Week", y = "Avg awake time after nap (hrs)",
       title = "Awake time between nap and bedtime, by week")

#centered 

graph_combo <- dataset4_average %>%
  pivot_longer(c(nap_centered, awake_centered),
               names_to = "measure", values_to = "hours") %>%
  mutate(measure = recode(measure,
                          nap_centered   = "Nap time",
                          awake_centered = "Awake time after nap"))


graph_combo|> 
  ggplot(aes(week, hours, color = measure)) +
  geom_line(linewidth = 0.9) + geom_point() +
  scale_x_continuous(breaks = seq(0, 52, 4)) +
  labs(x = "Week", y = "Average hours", color = NULL,
       title = "Average Nap Time and Awake Time After Nap, by Week") +
  theme_minimal() +
  theme(legend.position = "bottom")

Summary

Over the year, the toddler napped an average of 1.72 hours per day and stayed awake for an average of 4.82 hours between the end of the nap and bedtime. To compare weeks, I standardized both measures as z-scores centered at 0 on these yearly averages, so values above or below 0 mean a week was above or below 1.72 or 4.82 hours.

Over the course of the year, the two measures moved in opposite directions. Early in the year, weekly nap time was above the yearly average and awake time was below it. By the end of the year, nap time was below average and awake time was above it. From week to week, however, the relationship was not consistent. In some weeks, nap time and awake time were both above or both below average. Without a statistical test, it is hard to say whether the two are truly correlated.

#

#