NYC Flight Disruptions: Weather and Time of Day

Introduction:

This report examines and summarizes the flights departing from New York City in 2013 that were mostly affected by cancellations and departure delays longer than one hour. The analysis focuses on two possible explanations for these disruptions: severe weather and schedule time of departure.

Throughout this report, we will use the tidyverse package to create transformation and visualizations, the nycflights13 provides the flight data, and the DT package to display the data set.

library(tidyverse)
library(nycflights13)
library(DT)

The Data

The flights data set from nycflights13 package holds 336,776 individual flights that departed from three different New York Airports: John F. Kennedy International Airport (JFK), Newark Liberty International Airport (EWR), and LaGuardia Airport (LGA) during 2013. The dataset also contains 19 variables describing the information related to flights, such as, scheduled and actual departure times, delays, airlines, airports, and flight distances. The original information came from the U.S. Bureau of Transportation Statistics (https://nycflights13.tidyverse.org/reference/flights.html).

Variable	Description
`month`	Month of departure
`day`	Day of departure
`sched_dep_time`	Scheduled departure time
`dep_time`	Actual departure time
`dep_delay`	Departure delay in minutes; negative values indicate an early departure
`carrier`	Two-letter airline code
`flight`	Flight number
`origin`	Departure airport: EWR, JFK, or LGA
`dest`	Destination airport
`hour`	Scheduled departure hour
`dep_status`	Flight status created for this report based on cancellation and departure delay

Defining a Major Flight Disruption

flights_status <- flights %>%
  mutate(
    dep_status = case_when(
      is.na(dep_delay) ~ "canceled",
      dep_delay >60~ "delayed > 1 hour",
      dep_delay >0 ~ "delayed 1 hour or less",
      TRUE ~ "on time or early"
    )
  )

For this analysis, the major flight disruption is defined as either canceled flight or as a flight departing more than 60 minutes late.

Problematic Travel Days

problem_dates <- flights_status %>%
  group_by(month, day) %>%
  summarize(
    percentage =100 * mean(
      dep_status == "canceled" |
      dep_status == "delayed > 1 hour"
    ),
    count = n()
  ) %>%
  filter(percentage > 35) %>%
  arrange(desc(percentage))

problem_dates

## # A tibble: 12 × 4
## # Groups:   month [7]
##    month   day percentage count
##    <int> <int>      <dbl> <int>
##  1     2     9       61.3   684
##  2     3     8       58.6   979
##  3     2     8       54.4   930
##  4     5    23       44.0   988
##  5     7     1       42.1   966
##  6     9    12       40.7   992
##  7    12     5       39.7   969
##  8     6    28       37.1   994
##  9     7    23       36.3   997
## 10     7    10       36.3  1004
## 11     9     2       35.1   929
## 12     6    24       35.0   994

The analysis identified 12 different dates on which more than 35% of the scheduled flights were either canceled or delayed by more than one hour. The date that sticks out is February 9, where the highest disruption rate of 61.3% occurred. March 8 is the second highest disruption date; it had a reading of 58.6%. Lastly, February 8 had 54.4%, completing the top three most distrustful days in 2013.

Most of the dates contained around 900 to 1,000 scheduled flights, which suggests that these high disruption percentages are not being caused by extremely small samples.

Considering that on these dates several hundred flights were affected, it suggests that there were larger events that caused such disruption on airport operations, rather than being only a fraction of flights happening on those dates.

Weather and Other Causes of Disruptions

After 12 most disrupted dates were identified, I’ve done some research on the weather and other effects that occurred during each of the 12 days. Overall, the weather is connected to many of the dates well, while there are a few dates that appear to be affected directly by the airport operations and/or problems arising from the airline’s network.

Date	Weather or Other Event	Interpretation
February 8	Major blizzard	Severe weather - explains the disruptions.
February 9	Recovery from blizzard	Likely caused damage and required snow removals.
March 8	Winter storm	Severe weather - explains the disruptions.
May 23	Heavy rain, thunderstorms, and flash-flood warnings.	Severe weather - explains the disruptions.
June 24	Severe thunderstorms JFK runway construction.	Weather and airport operations both contributed to delays.
June 28	Heavy rain and flooding in the New York area.	Weather caused disruptions.
July 1	Thunderstorms and flash flooding.	Severe weather - explains the disruptions.
July 10	Rain and thunderstorms, no major weather event.	Weather may have contributed to delays but does give a clear cause of delays.
July 23	LaGuardia was recovering from an airplane landing-gear accident.	Airport operations - explains the disruptions.
September 2	Showers, thunderstorms, and possible flooding.	Weather contributed to the disruptions.
September 12	Strong winds, thunderstorms, and air-traffic restrictions.	Severe weather - explains the disruptions.
December 5	Major winter weather disrupted flights elsewhere in the U.S.	Disruptions elsewhere likely affected NYC flights.

Overall, the severe weather provides convincing evidence and explanation for many of the most disrupted dates, especially during the wintertime (February), followed by March snowstorms, then summer thunderstorms, and lastly September storms. However, my research also shows why it’s important not to assume that every high delay was caused by local weather. Runway construction, airport incidents, and weather disruptions elsewhere in the national flight network, all have contributed to flight delays.

Disruptions by Scheduled Departure Hour

hourly_delays <- flights_status %>%
  group_by(hour) %>%
  summarize(
    percentage = 100 * mean(
      dep_status == "canceled" |
      dep_status == "delayed > 1 hour"
    ),
    count = n()
  )

hourly_delays

## # A tibble: 20 × 3
##     hour percentage count
##    <dbl>      <dbl> <int>
##  1     1     100        1
##  2     5       1.89  1953
##  3     6       3.85 25951
##  4     7       3.47 22821
##  5     8       4.96 27242
##  6     9       5.16 20312
##  7    10       6.54 16708
##  8    11       6.56 16033
##  9    12       7.65 18181
## 10    13       8.92 19956
## 11    14      10.6  21706
## 12    15      12.0  23888
## 13    16      14.6  23002
## 14    17      14.7  24426
## 15    18      15.2  21783
## 16    19      18.7  21441
## 17    20      18.8  16739
## 18    21      20.1  10933
## 19    22      15.8   2639
## 20    23      10.1   1061

I’ve grouped the flights by their scheduled departure hour to see if the time of the day was related to the major disruptions. My idea was that certain (ex. morning) hours would cause and create more delays, since the staffing of the airport overnights is not the same as during the day or late hours. However, the percentages generally increase throughout the day, there is one very unusual observation at 1 AM local time, which will need to be investigated before the overall pattern interpretation.

The 1 A.M. Outlier

ggplot(
  data = hourly_delays,
  mapping = aes(x = hour, y = percentage)
) +
  geom_point() +
  geom_line()+
  labs(
    title = "Major Flight Disruptions by Scheduled Departure Hour",
    x = "Scheduled Departure Hour",
    y = "Canceled or Delayed More Than One Hour (%)"
  )

The 1 AM outlier is caused by extremely small number of observations. Only one flight in the entire data set was scheduled for that time, and that flight itself was canceled. Overall, this flight does not provide evidence that the 1 AM time is normally the riskiest departure time.

hourly_delays %>%
  arrange(desc(percentage))

## # A tibble: 20 × 3
##     hour percentage count
##    <dbl>      <dbl> <int>
##  1     1     100        1
##  2    21      20.1  10933
##  3    20      18.8  16739
##  4    19      18.7  21441
##  5    22      15.8   2639
##  6    18      15.2  21783
##  7    17      14.7  24426
##  8    16      14.6  23002
##  9    15      12.0  23888
## 10    14      10.6  21706
## 11    23      10.1   1061
## 12    13       8.92 19956
## 13    12       7.65 18181
## 14    11       6.56 16033
## 15    10       6.54 16708
## 16     9       5.16 20312
## 17     8       4.96 27242
## 18     6       3.85 25951
## 19     7       3.47 22821
## 20     5       1.89  1953

hour_outlier <- flights_status %>%
  filter(hour == 1)%>%
  select(
    month,
    day,
    sched_dep_time,
    dep_time,
    dep_delay,
    carrier,
    flight,
    origin,
    dest,
    distance,
    dep_status
  )

hour_outlier

## # A tibble: 1 × 11
##   month   day sched_dep_time dep_time dep_delay carrier flight origin dest 
##   <int> <int>          <int>    <int>     <dbl> <chr>    <int> <chr>  <chr>
## 1     7    27            106       NA        NA US        1632 EWR    LGA  
## # ℹ 2 more variables: distance <dbl>, dep_status <chr>

Looking at the observation itself, it confirms that this was a single US Airways flight from Newark to LaGuardia scheduled for 1:06 a.m. on July 27, and it was canceled.

The Regular Time-of-Day Pattern

hourly_delays_regular <- hourly_delays%>%
  filter(hour >= 5)

ggplot(
  data = hourly_delays_regular,
  mapping = aes(x = hour,y=percentage)
) +
  geom_point() +
  geom_line() +
  labs(
    title = "Major Flight Disruptions Throughout the Day",
    x = "Scheduled Departure Hour",
    y = "Canceled or Delayed More Than One Hour (%)"
  )

After concluding the misleading 1 AM observation, the relationship between the departure time and the disruption becomes clearer. The percentage of major disruption is only 1.9% at 5 AM, so morning hours are decent time to pick a fight with low disruptions. The percentage tends to increase thought the day, reaching 20.1% for flights scheduled at 9 PM. To me, this makes the evening flights the riskiest time for departure, if the goal is to avoid cancellations or delays for more than one hour.

My explanation for this shape is that the delays can build up thought the day. With so many flights and so many planes departing at specifically assigned times, just one delay early in the day, can cause disruption in the departure schedule. Factors such as weather, air-traffic restations, late departure, late arrival, all can accumulate and create flight delays.

Flights That Departed On Time on Disrupted Days

on_time_problem_dates <- flights_status %>%
  filter(
    (
      (month == 2 & day == 8) |
      (month == 2 & day == 9) |
      (month == 3 & day == 8) |
      (month == 5 & day == 23) |
      (month == 6 & day == 24) |
      (month == 6 & day == 28) |
      (month == 7 & day == 1) |
      (month == 7 & day == 10) |
      (month == 7 & day == 23) |
      (month == 9 & day == 2) |
      (month == 9 & day == 12) |
      (month == 12 & day == 5)
    ) &
    dep_status == "on time or early"
  ) %>%
  arrange(month, day, sched_dep_time)

on_time_problem_dates_display <- on_time_problem_dates %>%
  select(
    month,
    day,
    sched_dep_time,
    dep_time,
    dep_delay,
    carrier,
    flight,
    origin,
    dest
  )

datatable(on_time_problem_dates_display)

On 12 heavily disrupted dates, 3,441 flights were still able to depart on time or even early. Table above contains those flights and gives a comparable schedule departure time. My next step is to determine whether these successful flights were scheduled earlier on that day, before the worst event of the day or other problems got developed.

Testing the Morning-Flight Hypothesis

average_on_time_hour <- on_time_problem_dates %>%
  group_by(month, day) %>%
  summarize(
    avg_sched_hour = mean(hour),
    count = n()
  )

average_on_time_hour

## # A tibble: 12 × 4
## # Groups:   month [7]
##    month   day avg_sched_hour count
##    <int> <int>          <dbl> <int>
##  1     2     8           8.86   219
##  2     2     9          17.0    125
##  3     3     8          10.2    146
##  4     5    23           9.45   329
##  5     6    24           9.82   369
##  6     6    28          10.1    309
##  7     7     1           9.43   229
##  8     7    10           9.60   360
##  9     7    23          10.0    257
## 10     9     2           9.85   386
## 11     9    12           9.03   386
## 12    12     5          10.1    326

The results show and support the idea that the earlier flights were more likely to evade the disruptions that occurred later on the day. On 11 out of 12 problematic dates, the average scheduled departure hour of flights that left early was between 8.9 and 10.2. February 9 is the only exception, with an average scheduled departure hour of 16.96, or at around 5 PM.

all_on_time_flights <- flights_status %>%
  filter(dep_status == "on time or early")

mean(all_on_time_flights$hour)

## [1] 12.27121

As a comparison, the average scheduled hour for all flights that departed on time was about shortly after noon. Since the flights that successfully departed on those problematic days averaged 9 to 10 AM, provide additional evidence that the morning flights are more likely to leave before the day’s disruption becomes more severe.

The February 9 Exception

February 9 stands out as an exception because the flights that departed successfully averaged almost 5 PM, which is very different from other problematic dates. This date occurred immediately after the big blizzard. JFK and LaGuardia began resuming their operations for some flights around 9 AM, while the commercial departures from Newark did not begin until noon, so the airlines counited operating on reduced schedules. Therefore, the afternoon average makes sense since many morning flights were unable to operate while the airports were still clearing the snow from runways and recovering their operations from the storm.

Conclusion

Overall, this analysis shows that the major flight disruptions in New York during 2013 were related to both specific events and the time of day. The twelve dates had disruption rates above 35%, and further analysis of them showed that the severe storms, flooding, weather, airport operation, and other aviation events explain the causes of these unusually bad days. Time-of-day analysis also showed a clear pattern, where major disruptions generally become more common later in the day, rising from less than 2% at 5 AM, to about 20% at 9 PM. Flights that successfully departed on the most disrupted days usually had early departures in the morning, yet again this supports the idea that the earlier flights had a better chance of leaving before the problems accumulated. Overall, both weather and the accumulation of delays thought the day appeared to be an important part in flight departure reliability.

Sources

https://www.weather.gov/okx/storm02092013 https://www.cbsnews.com/newyork/news/limited-flights-resume-at-jfk-laguardia-and-newark-airports/ https://www.cbsnews.com/newyork/news/winter-storm-blankets-much-of-tri-state-in-heavy-wet-snow/ https://www.cbsnews.com/newyork/news/severe-weather-on-tap-for-tri-state-thursday/ https://www.cbsnews.com/newyork/news/severe-weather-threat-prompts-flight-delays-of-almost-4-hours/ https://www.weather.gov/aly/MajorFloods https://vinnews.com/2013/07/01/new-york-weather-causes-huge-delays-at-nyc-area-airports/ https://www.ntsb.gov/investigations/Pages/DCA13FA131.aspx https://www.bts.gov/newsroom/airlines-report-13-tarmac-delays-over-three-hours-domestic-flights-three-tarmac-delays https://www.theyeshivaworld.com/news/nyc/184099/nyc-showers-thunderstorms-through-tuesday-morning.html https://www.weather.gov/fwd/december72013 https://www.bts.gov/topics/airlines-and-airports/understanding-reporting-causes-flight-delays-and-cancellations?os=___