Introduction

What is the relationship between flight cancellations and various temporal factors for 2013 flights out of the three major airports serving New York City?

With the flights data set obtained from the Bureau of Transportation Statistics: https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236, this report intends to answer the question posed above. Data set flights contains 336,776 observations of flights from the airports serving New York City in 2013. There are 19 variables but the most pertinent to this project’s scope are month and day (the calendar information for a flight’s departure), sched_dep_time, dep_time, and dep_delay which are the time a flight was scheduled to depart, the time it truly departed, and the difference between the two. Those times are listed as three or four digit, base ten numbers. hour and minute are the individual numbers for each flights scheduled departure time. Further variables cover destinations, airline carriers, and other details that do not contribute to this research question.

flights is pulled from the r library nycflights13 and visuals are made through the tidyverse library. For a smaller data set, the DT library could be used to make a data table of flights but the immense quantity of observations makes it less than ideal. DT will still be used for other instances of data tables.

library(tidyverse)
library(nycflights13)
library(DT)

Worst Days for Cancelled or Delayed Flights

A logical first place to start when determining the relationship between flight delays and time is to identify the days with the highest rates of cancelled or delayed flights. Using 35% as the minimum threshold, a data set will be compiled from flights to highlight the days in which high rates of the flights where cancelled (lacking a departure time) or had a departure delay of at least 60 minutes.

flights_by_day1 <- flights %>%
  group_by(month, day) %>%
  summarize(count = n(),
            perc_del_can = 100 * mean(dep_delay >= 60 | is.na(dep_time))) %>%
  select(month, day, perc_del_can, count) %>%
  filter(perc_del_can >= 35)
datatable(flights_by_day1, options = list(scrollX = TRUE))

The newly calculated variable perc_del_can relays the percentage of flights in a single day that were delayed an hour or more or cancelled. Also included in the table is the variable count which is the total number of flights scheduled that day. It is revealed there were 12 days in 2013 on which at least 35% of the flights experienced lengthy delays or cancellations. In an effort to explain these occurrences, brief Google searches will be conducted for the dates in the table above.

February 8 and 9: On these days each experiencing high rates of delays and cancellations, a blizzard passed through New York. Source: https://www.weather.gov/okx/storm02092013

March 8: A month after the previous blizzard, another one hit NYC leading to 59% of flights being cancelled or delayed at least an hour. Source: https://www.nbcnews.com/news/photo/snow-storm-blankets-new-york-city-leaving-pretty-scenes-sloshy-flna1c8776255

May 23: A heavy rain event, featuring thunderstorms hit New York on this day. Source: https://deploy.weatherspark.com/h/m/23912/2013/5/Historical-Weather-in-May-2013-in-New-York-City-New-York-United-States

June 24: This day had forecasted severe thunderstorms, although they seem to have missed NYC. Source: https://www.cbsnews.com/newyork/news/severe-weather-threat-prompts-flight-delays-of-almost-4-hours/

June 28: Although NYC had favorable weather, a storm system to the north could have played a role in the flight delays and cancellations. Source: https://www.weather.gov/aly/july2026flashflood

July 1: There was some rain and fog present in New York. Source: https://www.timeanddate.com/weather/usa/new-york/historic?month=7&year=2013

July 10: A bit of rainfall and haze, although seemingly not much. Source: https://www.timeanddate.com/weather/usa/new-york/historic?month=7&year=2013

July 23: Rain and thunderstorms were present on this day in NYC.Source: https://weatherspark.com/h/m/23912/2013/7/Historical-Weather-in-July-2013-in-New-York-City-New-York-United-States

September 2: Fog and rain were present at different times throughout the day. Source: https://www.timeanddate.com/weather/usa/new-york/historic?month=9&year=2013

September 12: Rain and fog made appearances throughout the day. Source: https://www.timeanddate.com/weather/usa/new-york/historic?month=9&year=2013

December 5: This day saw some rain with lots of fog. Source: https://weatherspark.com/h/m/23912/2013/12/Historical-Weather-in-December-2013-in-New-York-City-New-York-United-States#google_vignette

Based on the above observations, there is some weather component that can explain most if not all of the days with high rates of delayed or cancelled flights.

Cancelled or Delayed Flights by Hour

Having looked at the worst days for flying out of New York in 2013, it may now be useful to explore the worst times of day to fly out. The following data table summarizes the flight delay cancellation data by hour of day.

flights_by_hour1 <- flights %>%
  group_by(hour) %>%
  summarize(count = n(),
            perc_del_canc = 100 * mean(dep_delay >= 60 | is.na(dep_time))) %>%
  select(hour, perc_del_canc, count)
datatable(flights_by_hour1, options = list(scrollX = TRUE))

The variable perc_del_canc shows the percentage of flights delayed by an hour or more or cancelled for each hour by scheduled departure time. It is worth pointing out hour 1 which only had a single flight scheduled all year and is perhaps not fair to include as a comparison. Outside of that, hour 5 had the best rate with only 1.95% of flights cancelled or delayed by 60+ minutes, while hour 21 had the worst rate, with 20.36% having been cancelled or delayed greatly.

Another way of visualizing this data to make it more digestible is by plotting the relationship. With hour as the x variable and perc_del_canc as the y variable, it yields the plot below.

ggplot(data = flights_by_hour1, mapping = aes(x = hour, y = perc_del_canc)) +
  geom_point() +
  geom_smooth()  +
  labs(x = "hour",
       y = "percentage of flights cancelled or delayed 60+ min",
       title = "2013 NYC flight cancellation or delay % by hour",
       caption = "Data obtained from Bureau of Transportation Stats")

The presence of the lone flight scheduled for a departure in hour 1 and yielded a 100% cancellation or delay rate is undoubtedly an outlier here. The margin of error for the curve fails to contain multiple data points at hours 5 and 6 currently. Removing this outlier will allow for clearer interpretation of the plot and a better understanding of the relationship.

flights_by_hour2 <- flights_by_hour1 %>%
  filter(hour > 1)
ggplot(data = flights_by_hour2, mapping = aes(x = hour, y = perc_del_canc)) +
  geom_point() +
  geom_smooth()  +
  labs(x = "hour",
       y = "percentage of flights cancelled or delayed 60+ min",
       title = "2013 NYC flight cancellation or delay % by hour",
       subtitle = "Outlier at hour 1 removed",
       caption = "Data obtained from Bureau of Transportation Stats")

This new plot without the outlier produces a much more clear relationship. The rest of the data does not appear to have any outliers since the counts are all far more robust. There is a noticeable rise in the rate of flight cancellations and long delays as the day progresses from morning into afternoon and then into the evening. After hour 21, that rate drops steeply but does not quite reach the levels of the early morning. This shape likely has a few different factors behind it. The early morning flights are presumably taking off at a time when fewer planes are landing than in the afternoon, which leads to less competition for runways. Additionally, an early morning flight will be that plane’s first flight of the day and will not have compounded a delay from an earlier flight in the same way that a plane doing its second flight in the evening may experience. Furthermore, if there is adverse weather, there is the option of moving an early flight time up so that it gets out before the weather which is not a delay. Later in the day, that option does not exist since there are already flights scheduled throughout the day. As a result, the riskiest time of day for delays or cancellations comes after 7pm and through the hour of 9pm. These times had rates of cancellations or long delays all above 18%.

Early and On-time Flights from the Worst Days

Although the 12 days identified in the first section had the lowest rates of early or on-time departures from the year, there were still many flights that got out without delay those days. The following data table holds the data set of all early and on-time flight departures from 2013’s days with 35% or higher rates of long delays and cancellations.

flights_early_worst_days <- flights %>%
  filter(dep_delay <= 0 & 
           ((month == 2 & (day == 8 | day == 9)) |
              (month == 3 & day == 8) |
              (month == 5 & day == 23) |
              (month == 6 & (day == 24 | day == 28)) |
              (month == 7 & (day == 1 | day == 10 | day == 23)) |
              (month == 9 & (day == 2 | day == 12)) |
              (month == 12 & day == 5))) %>%
  select(month,day,dep_time:dep_delay,hour)
datatable(flights_early_worst_days, options = list(scrollX = TRUE))

Across those 12 worst days for NYC flights, 3,441 flights still managed to leave on time or early. Based on the plot seen in the previous section, it may be reasonable to hypothesize that most of these flights are from the morning. After all, that is when the cancellation and delay rates were lowest in 2013.

To get a measure of this, the average hour of scheduled departure will be taken for each day from the 12.

flights_early_worst_days_avg <- flights_early_worst_days %>%
  group_by(month, day) %>%
  summarize(count = n(),
            avg_dep_hr = mean(hour)) %>%
  select(month, day, avg_dep_hr, count)
datatable(flights_early_worst_days_avg, options = list(scrollX = TRUE))

This calculation of the average hour in the form of the new variable avg_dep_hr reveals that on 11 of the 12 highest days of cancellations, the average scheduled departure time for planes that left on time or early was before 11am. The one exception to that was February 9, which we know to be the second day of a blizzard. With an average departure hour of nearly 5pm, it is very possible the blizzard was over by the afternoon so evening flights had more success than morning flights that day.

Conclusion

Through analysis of the flights data set, there is evidence to suggest flight cancellations and lengthy delays are more common in the afternoon and evening than they are in the morning. In the cases of days with high rates of flight cancellations and delays exceeding an hour, weather will almost always be a contributing factor.