Exploratory Data Analysis in R: Airline Delays

knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
library(dplyr)
library(ggplot2)
library(nycflights13)

Introduction

EDA is not about finding one final answer, but about asking questions, creating summaries and visualizations, and then using what we discover to ask new questions.

Let’s pretend we’re analysts for a large airline or we work in airport operations at a large airport to:

• Airlines want to reduce delays (lower costs, happier customers))
• Travelers want to know which flights are more reliable.
• Airports want to manage congestion.

We will use the flights dataset from the nycflights13 package, which contains all flights departing New York City in 2013.

---

Step 1: Take a Look at the Data

flights

---

Step 2: Overall Flight Delays

flights %>%
  summarize(
    avg_dep_delay = mean(dep_delay, na.rm = TRUE),
    median_dep_delay = median(dep_delay, na.rm = TRUE),
    n_flights = n()
  )

Discussion:

• The average delay is much higher than the median delay.
• Most flights are only slightly late, but a few extremely delayed flights skew the average.

flights %>%
  ggplot(aes(x = dep_delay)) +
  geom_histogram(binwidth = 5, fill = "steelblue", color = "white") +
  xlim(-50, 200) +
  labs(title = "Distribution of Departure Delays",
       x = "Departure Delay (minutes)", y = "Number of Flights")

Business Interpretation:

• For airlines: extreme delays represent operational failures that are costly and need special focus.
• For passengers: averages can be misleading — most flights are fine, but some ruin the experience.

---

Step 3: Do Delays Differ by Airline?

flights %>%
  group_by(carrier) %>%
  summarize(avg_dep_delay = mean(dep_delay, na.rm = TRUE)) %>%
  ggplot(aes(x = reorder(carrier, avg_dep_delay), y = avg_dep_delay)) +
  geom_col(fill = "darkred") +
  coord_flip() +
  labs(title = "Average Departure Delay by Airline",
       x = "Airline", y = "Average Delay (minutes)")

Discussion:

• Some airlines consistently perform better than others.

Business Interpretation:

• Airlines: compare performance against competitors.
• Travelers: choose carriers with stronger on-time records.

---

Step 4: Do Delays Change by Time of Day?

flights %>%
  mutate(hour = sched_dep_time %/% 100) %>%
  group_by(hour) %>%
  summarize(avg_dep_delay = mean(dep_delay, na.rm = TRUE)) %>%
  ggplot(aes(x = hour, y = avg_dep_delay)) +
  geom_line(color = "blue", linewidth = 1.2) +
  geom_point() +
  labs(title = "Average Departure Delay by Hour of Day",
       x = "Scheduled Departure Hour", y = "Average Delay (minutes)")

Discussion:

• Delays are low in the morning but increase steadily throughout the day.

Business Interpretation:

• Airlines: add buffer in evening schedules to avoid knock-on effects.
• Travelers: fly early to avoid delays.

---

Step 5: From EDA to Prediction

So far, our EDA has revealed patterns:

• Delays are not evenly distributed (most flights okay, a few very late).
• Some airlines perform better than others.
• Time of day strongly influences delays.

This naturally suggests we could predict departure delay using some of these variables. For example:

• Predictor: carrier → Some airlines are more punctual.
• Predictor: hour → Morning vs evening matters.
• Predictor: month → Seasonal weather patterns could matter.

Let’s create a simplified dataset ready for modeling:

flights_model <- flights %>%
  mutate(
    hour = sched_dep_time %/% 100,
    delayed = ifelse(dep_delay > 15, 1, 0)   # classify as delayed (>15 min)
  ) %>%
  select(carrier, month, hour, delayed) %>%
  na.omit()

head(flights_model)

Discussion: Here we transformed the problem into a classification task:

• Outcome variable: delayed (1 if departure delay > 15 minutes, else 0).
• Predictors: carrier, month, hour.

We are not building a model here, but after EDA we can be confident that these variables carry predictive power because:

• carrier differences are visible in averages.
• hour shows a clear upward trend in delays.
• month likely captures seasonal/weather effects (e.g., winter storms).

Business Interpretation:

• Airlines could use such a model to anticipate delays and reallocate resources.
• Airports could predict peak risk periods and staff accordingly.
• Passengers (via apps) could be warned of flights with higher delay probabilities.

---

Step 6: Conclusion

In this notebook, we saw how EDA is an iterative process:

1. Start with overall patterns (delays in general).
2. Drill down by categories (airline).
3. Explore additional dimensions (time of day).
4. Connect findings to predictive modeling: identifying useful predictors.

EDA is not the end — it’s the foundation for building models that can help answer “what will happen?” instead of just “what happened?”.

---