Flight Delay Analysis and Prediction (2019–2023)
SHEN KE(24077610) ,WANG YING(24075937), HOU YAHUI(24051848), LI JIAWEI(24072994), GONG WEILIN(24075383)
2025-05-27
# 1.Introduction
Flight delays represent a critical operational challenge in civil aviation, disrupting passenger schedules and imposing multifaceted cost implications on airlines – encompassing fuel inefficiencies, crew management complexities, and brand reputation risks. FAA analyses indicate that the sector incurs annual economic losses exceeding billions of U.S. dollars, with systemic propagation effects amplifying impacts: primary delays frequently trigger cascading network disruptions. This domino effect becomes particularly pronounced during peak operational periods, exacerbating scheduling inefficiencies across subsequent flight rotations.
To mitigate these challenges, we propose developing a predictive modeling framework to anticipate flight delays. Our methodology incorporates systematic data preprocessing methodologies addressing missing values, temporal normalization through lubridate-based standardization, and strategic dataset filtering to isolate operational flights. Feature engineering identified critical temporal and operational predictors including departure time windows, weekday operational patterns, and airport-specific identifiers. We evaluated both linear regression and XGBoost architectures through caret-enabled cross-validation, with the gradient boosting framework demonstrating superior predictive capability. The optimized gradient boosting architecture demonstrated robust performance in estimating departure delay durations through effective utilization of engineered features.
# 2.Objective
This project addresses two key data science problems in the context of flight operations:
Regression Problem: Build a predictive model to estimate the departure delay (in minutes) of a flight, based on features such as scheduled time, origin and destination airports, airline, day of week, and distance.
Exploratory Causal Analysis: Analyze and compare different delay causes (e.g., weather, carrier, security, NAS system, late aircraft) and their contributions to overall delay time. We aim to understand which factors are most influential and how these vary across airlines, airports, and seasons.
# 3.Flight Data Cleaning and Feature Engineering
3.1. Read and Inspect the Raw Flight Dataset
This section reads the original CSV file and performs initial data inspection to understand the structure.
# Read the CSV file
flights <- read.csv("flights_sample_3m.csv", stringsAsFactors = FALSE)
# Preview the first few rows
head(flights)## FL_DATE AIRLINE AIRLINE_DOT AIRLINE_CODE
## 1 2019-01-09 United Air Lines Inc. United Air Lines Inc.: UA UA
## 2 2022-11-19 Delta Air Lines Inc. Delta Air Lines Inc.: DL DL
## 3 2022-07-22 United Air Lines Inc. United Air Lines Inc.: UA UA
## 4 2023-03-06 Delta Air Lines Inc. Delta Air Lines Inc.: DL DL
## 5 2020-02-23 Spirit Air Lines Spirit Air Lines: NK NK
## 6 2019-07-31 Southwest Airlines Co. Southwest Airlines Co.: WN WN
## DOT_CODE FL_NUMBER ORIGIN ORIGIN_CITY DEST DEST_CITY
## 1 19977 1562 FLL Fort Lauderdale, FL EWR Newark, NJ
## 2 19790 1149 MSP Minneapolis, MN SEA Seattle, WA
## 3 19977 459 DEN Denver, CO MSP Minneapolis, MN
## 4 19790 2295 MSP Minneapolis, MN SFO San Francisco, CA
## 5 20416 407 MCO Orlando, FL DFW Dallas/Fort Worth, TX
## 6 19393 665 DAL Dallas, TX OKC Oklahoma City, OK
## CRS_DEP_TIME DEP_TIME DEP_DELAY TAXI_OUT WHEELS_OFF WHEELS_ON TAXI_IN
## 1 1155 1151 -4 19 1210 1443 4
## 2 2120 2114 -6 9 2123 2232 38
## 3 954 1000 6 20 1020 1247 5
## 4 1609 1608 -1 27 1635 1844 9
## 5 1840 1838 -2 15 1853 2026 14
## 6 1010 1237 147 15 1252 1328 3
## CRS_ARR_TIME ARR_TIME ARR_DELAY CANCELLED CANCELLATION_CODE DIVERTED
## 1 1501 1447 -14 0 0
## 2 2315 2310 -5 0 0
## 3 1252 1252 0 0 0
## 4 1829 1853 24 0 0
## 5 2041 2040 -1 0 0
## 6 1110 1331 141 0 0
## CRS_ELAPSED_TIME ELAPSED_TIME AIR_TIME DISTANCE DELAY_DUE_CARRIER
## 1 186 176 153 1065 NA
## 2 235 236 189 1399 NA
## 3 118 112 87 680 NA
## 4 260 285 249 1589 0
## 5 181 182 153 985 NA
## 6 60 54 36 181 141
## DELAY_DUE_WEATHER DELAY_DUE_NAS DELAY_DUE_SECURITY DELAY_DUE_LATE_AIRCRAFT
## 1 NA NA NA NA
## 2 NA NA NA NA
## 3 NA NA NA NA
## 4 0 24 0 0
## 5 NA NA NA NA
## 6 0 0 0 0# Display structure of the dataset
str(flights)## 'data.frame': 3000000 obs. of 32 variables:
## $ FL_DATE : chr "2019-01-09" "2022-11-19" "2022-07-22" "2023-03-06" ...
## $ AIRLINE : chr "United Air Lines Inc." "Delta Air Lines Inc." "United Air Lines Inc." "Delta Air Lines Inc." ...
## $ AIRLINE_DOT : chr "United Air Lines Inc.: UA" "Delta Air Lines Inc.: DL" "United Air Lines Inc.: UA" "Delta Air Lines Inc.: DL" ...
## $ AIRLINE_CODE : chr "UA" "DL" "UA" "DL" ...
## $ DOT_CODE : int 19977 19790 19977 19790 20416 19393 19805 20452 20416 19930 ...
## $ FL_NUMBER : int 1562 1149 459 2295 407 665 2134 4464 590 223 ...
## $ ORIGIN : chr "FLL" "MSP" "DEN" "MSP" ...
## $ ORIGIN_CITY : chr "Fort Lauderdale, FL" "Minneapolis, MN" "Denver, CO" "Minneapolis, MN" ...
## $ DEST : chr "EWR" "SEA" "MSP" "SFO" ...
## $ DEST_CITY : chr "Newark, NJ" "Seattle, WA" "Minneapolis, MN" "San Francisco, CA" ...
## $ CRS_DEP_TIME : int 1155 2120 954 1609 1840 1010 1010 1643 530 2125 ...
## $ DEP_TIME : num 1151 2114 1000 1608 1838 ...
## $ DEP_DELAY : num -4 -6 6 -1 -2 147 -9 -6 -3 -9 ...
## $ TAXI_OUT : num 19 9 20 27 15 15 23 22 11 19 ...
## $ WHEELS_OFF : num 1210 2123 1020 1635 1853 ...
## $ WHEELS_ON : num 1443 2232 1247 1844 2026 ...
## $ TAXI_IN : num 4 38 5 9 14 3 8 41 8 3 ...
## $ CRS_ARR_TIME : int 1501 2315 1252 1829 2041 1110 1159 1945 717 2355 ...
## $ ARR_TIME : num 1447 2310 1252 1853 2040 ...
## $ ARR_DELAY : num -14 -5 0 24 -1 141 -29 23 -11 1 ...
## $ CANCELLED : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CANCELLATION_CODE : chr "" "" "" "" ...
## $ DIVERTED : num 0 0 0 0 0 0 0 0 0 0 ...
## $ CRS_ELAPSED_TIME : num 186 235 118 260 181 60 109 122 227 210 ...
## $ ELAPSED_TIME : num 176 236 112 285 182 54 89 151 219 220 ...
## $ AIR_TIME : num 153 189 87 249 153 36 58 88 200 198 ...
## $ DISTANCE : num 1065 1399 680 1589 985 ...
## $ DELAY_DUE_CARRIER : num NA NA NA 0 NA 141 NA 0 NA NA ...
## $ DELAY_DUE_WEATHER : num NA NA NA 0 NA 0 NA 0 NA NA ...
## $ DELAY_DUE_NAS : num NA NA NA 24 NA 0 NA 23 NA NA ...
## $ DELAY_DUE_SECURITY : num NA NA NA 0 NA 0 NA 0 NA NA ...
## $ DELAY_DUE_LATE_AIRCRAFT: num NA NA NA 0 NA 0 NA 0 NA NA ...3.2. Load Required Libraries
# View column names
colnames(flights)## [1] "FL_DATE" "AIRLINE"
## [3] "AIRLINE_DOT" "AIRLINE_CODE"
## [5] "DOT_CODE" "FL_NUMBER"
## [7] "ORIGIN" "ORIGIN_CITY"
## [9] "DEST" "DEST_CITY"
## [11] "CRS_DEP_TIME" "DEP_TIME"
## [13] "DEP_DELAY" "TAXI_OUT"
## [15] "WHEELS_OFF" "WHEELS_ON"
## [17] "TAXI_IN" "CRS_ARR_TIME"
## [19] "ARR_TIME" "ARR_DELAY"
## [21] "CANCELLED" "CANCELLATION_CODE"
## [23] "DIVERTED" "CRS_ELAPSED_TIME"
## [25] "ELAPSED_TIME" "AIR_TIME"
## [27] "DISTANCE" "DELAY_DUE_CARRIER"
## [29] "DELAY_DUE_WEATHER" "DELAY_DUE_NAS"
## [31] "DELAY_DUE_SECURITY" "DELAY_DUE_LATE_AIRCRAFT"3.3. Format Dates and Create Temporal Features
Convert flight dates to proper date format and create features such as weekday and US federal holiday flags.
flights <- flights %>%
mutate(
FL_DATE = as.Date(FL_DATE), # Convert to Date format
Weekday = wday(FL_DATE, label = TRUE, abbr = FALSE), # Extract day of the week
IsHoliday = isHoliday(as.timeDate(FL_DATE), holiday = "USFederal") # Flag US federal holidays
)3.4. Define Helper Function to Convert Time Format
This function converts HHMM time values into decimal hours for easier calculation.
convert_time <- function(x) {
x <- as.numeric(x) # Convert to numeric
hour <- x %/% 100 # Extract hour
minute <- x %% 100 # Extract minutes
return(hour + minute / 60) # Convert to decimal hour
}3.5. Apply Time Conversion and Calculate Delay Metrics
This section uses the helper function to create features like flight duration and delays.
flights <- flights %>%
mutate(
DepHour = convert_time(DEP_TIME), # Actual departure time
CRSDepHour = convert_time(CRS_DEP_TIME), # Scheduled departure time
ArrHour = convert_time(ARR_TIME), # Actual arrival time
CRSArrHour = convert_time(CRS_ARR_TIME), # Scheduled arrival time
FlightTime = ArrHour - DepHour, # Total flight time
DepDelay = DepHour - CRSDepHour, # Departure delay
ArrDelay = ArrHour - CRSArrHour # Arrival delay
)3.6. Convert Categorical Columns to Factor
This step prepares the categorical variables for modeling or analysis.
flights <- flights %>%
mutate(
AIRLINE = as.factor(AIRLINE),
ORIGIN = as.factor(ORIGIN),
DEST = as.factor(DEST)
)3.7. Inspect Distance Distribution
Get a statistical summary of the flight distance.
summary(flights$DISTANCE)## Min. 1st Qu. Median Mean 3rd Qu. Max.
## 29.0 377.0 651.0 809.4 1046.0 5812.03.8. Create Binary Delay Label
Create a new feature indicating whether the flight was delayed upon arrival.
flights <- flights %>%
mutate(Delayed = ifelse(ArrDelay > 0.25, "Delayed", "OnTime")) %>%
mutate(Delayed = as.factor(Delayed))3.9. Normalize Continuous Features
Standardize continuous variables like distance and flight time for later use in modeling.
flights <- flights %>%
mutate(
DISTANCE_SCALED = scale(DISTANCE),
FLIGHTTIME_SCALED = scale(FlightTime)
)3.10. Analyze Cancellations by Weekday
Count the number of cancelled flights grouped by the day of the week.
Sys.setlocale("LC_TIME", "C")## [1] "C"flights$Weekday <- weekdays(as.Date(flights$FL_DATE))
library(knitr)
flights %>%
filter(CANCELLED == 1) %>%
group_by(Weekday) %>%
summarise(CancelledFlights = n()) %>%
arrange(match(Weekday, c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"))) %>%
kable(caption = "Cancelled Flights by Weekday")| Weekday | CancelledFlights |
|---|---|
| Monday | 11939 |
| Tuesday | 10294 |
| Wednesday | 11356 |
| Thursday | 12655 |
| Friday | 11605 |
| Saturday | 9645 |
| Sunday | 11646 |
3.11. Analyze Cancellation Rate by Holiday and Weekday
Summarize flight cancellations by holiday and by weekday, calculating cancellation rates.
holiday_cancel_summary <- flights %>%
group_by(IsHoliday) %>%
summarise(
TotalFlights = n(),
CancelledFlights = sum(CANCELLED),
CancelRate = round(100 * CancelledFlights / TotalFlights, 2)
)
kable(holiday_cancel_summary, caption = "Cancellation Rate by Holiday")| IsHoliday | TotalFlights | CancelledFlights | CancelRate |
|---|---|---|---|
| FALSE | 2179216 | 57849 | 2.65 |
| TRUE | 820784 | 21291 | 2.59 |
weekday_cancel_summary <- flights %>%
group_by(Weekday) %>%
summarise(
TotalFlights = n(),
CancelledFlights = sum(CANCELLED),
CancelRate = round(100 * CancelledFlights / TotalFlights, 2)
) %>%
arrange(match(Weekday, c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")))
kable(weekday_cancel_summary, caption = "Cancellation Rate by Weekday")| Weekday | TotalFlights | CancelledFlights | CancelRate |
|---|---|---|---|
| Monday | 446600 | 11939 | 2.67 |
| Tuesday | 416562 | 10294 | 2.47 |
| Wednesday | 422837 | 11356 | 2.69 |
| Thursday | 446925 | 12655 | 2.83 |
| Friday | 446292 | 11605 | 2.60 |
| Saturday | 384223 | 9645 | 2.51 |
| Sunday | 436561 | 11646 | 2.67 |
3.12. Save the Cleaned Dataset
Limit data to 1 million rows and export to CSV and Excel formats.
# Keep only the first 1 million rows
flights <- head(flights, 1000000)
# Save as CSV file
write.csv(flights, "D:/UM/第一学期课程/大数据编程R004/group/本体/新建文件夹/flights_cleaned002.csv", row.names = FALSE)
# Install and load openxlsx for Excel output
#install.packages("openxlsx")
library(openxlsx)
# Save as Excel file
write.xlsx(flights, "D:/UM/第一学期课程/大数据编程R004/group/本体/新建文件夹/flights_cleaned.xlsx")# 4. EDA
Data Loading and Preprocessing
# Load data
flights <- read.csv("flights_cleaned002.csv")
flights <- flights %>%
mutate(
FL_DATE = as.Date(FL_DATE),
DEP_HOUR = floor(CRS_DEP_TIME/100),
WEEKDAY = wday(FL_DATE, label = TRUE, abbr = TRUE, locale = "C"),
MONTH_NUM = month(FL_DATE),
MONTH = factor(MONTH_NUM,
levels = 1:12,
labels = c("Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"),
ordered = TRUE)
)4.1. Basic Statistical Analysis
4.1.1 Distribution of Departure Delays
ggplot(flights, aes(x = DEP_DELAY)) +
geom_histogram(bins = 50, fill = "steelblue", alpha = 0.7) +
labs(title = "Distribution of Departure Delays",
x = "Departure Delay (minutes)",
y = "Count") +
theme_minimal() +
scale_x_continuous(limits = c(-60, 180))
The histogram demonstrates that the vast majority of flights depart on time or with only minor delays, as shown by the sharp peak at zero. However, the distribution is heavily right-skewed, with a long tail extending towards significant positive delays. This indicates that while most flights are punctual, a small subset experiences severe delays, which can have a disproportionate impact on overall passenger satisfaction and operational efficiency.
4.1.2 Boxplot of Departure Delays by Airline
ggplot(flights, aes(x = AIRLINE_CODE, y = DEP_DELAY)) +
geom_boxplot(fill = "lightblue", alpha = 0.7) +
labs(title = "Departure Delays by Airline",
x = "Airline Code",
y = "Departure Delay (minutes)") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
scale_y_continuous(limits = c(-30, 120))
The boxplot reveals that most airlines have a similar median departure delay, but there are notable differences in the spread and presence of outliers. Several airlines exhibit a wider interquartile range and a higher frequency of extreme delay values, suggesting inconsistency in their operational performance. The presence of many outliers across all airlines highlights the challenge of managing rare but severe delay events.
4.1.3 Average Departure Delay by Origin Airport
flights %>%
group_by(ORIGIN) %>%
summarise(avg_delay = mean(DEP_DELAY, na.rm = TRUE)) %>%
arrange(desc(avg_delay)) %>%
head(20) %>%
ggplot(aes(x = reorder(ORIGIN, avg_delay), y = avg_delay)) +
geom_bar(stat = "identity", fill = "steelblue", alpha = 0.7) +
labs(title = "Top 20 Origin Airports by Average Departure Delay",
x = "Origin Airport",
y = "Average Departure Delay (minutes)") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
coord_flip()
This horizontal bar chart highlights that a few origin airports, such as PSM and FOD, have exceptionally high average departure delays, far exceeding the rest. The majority of airports in the top 20 have average delays between 15 and 40 minutes, but the top one or two airports stand out as significant outliers. This suggests that targeted interventions at these specific airports could yield substantial improvements in overall delay statistics.
4.1.4 Departure Delay vs. Flight Distance
flights_sample <- flights %>%
sample_n(10000)
ggplot(flights_sample, aes(x = DISTANCE, y = DEP_DELAY)) +
geom_point(alpha = 0.1, color = "steelblue") +
geom_smooth(method = "loess", color = "red") +
labs(title = "Departure Delay vs. Flight Distance",
x = "Flight Distance (miles)",
y = "Departure Delay (minutes)") +
theme_minimal() +
scale_y_continuous(limits = c(-30, 120))
The scatter plot, complemented by a LOESS trend line, shows that there is no strong linear relationship between flight distance and departure delay. Delays are widely distributed across all distances, with a dense cluster of short delays and a few extreme values at all distance ranges. The trend line remains relatively flat, indicating that factors other than distance—such as airport congestion or weather—are more influential in causing delays.
4.2. Delay Cause Analysis
4.2.1 Pie Chart of Delay Causes
delay_causes <- flights %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
pivot_longer(everything(), names_to = "Cause", values_to = "Count")
ggplot(delay_causes, aes(x = "", y = Count, fill = Cause)) +
geom_bar(stat = "identity", width = 1) +
coord_polar("y", start = 0) +
labs(title = "Proportion of Delay Causes",
fill = "Delay Cause") +
theme_minimal() +
theme(axis.text = element_blank(),
axis.title = element_blank()) +
scale_fill_viridis_d()
The pie chart clearly shows that carrier-related and late aircraft delays are the dominant causes, together accounting for the majority of delay minutes. NAS (National Airspace System) and weather delays are less significant, while security-related delays are minimal. This distribution suggests that operational and scheduling inefficiencies within airlines are the primary drivers of delays, rather than external factors.
4.2.2 Correlation Heatmap of Delay Variables
delay_vars <- flights %>%
select(DEP_DELAY, ARR_DELAY, TAXI_OUT, TAXI_IN,
DELAY_DUE_CARRIER, DELAY_DUE_WEATHER, DELAY_DUE_NAS,
DELAY_DUE_SECURITY, DELAY_DUE_LATE_AIRCRAFT)
cor_matrix <- cor(delay_vars, use = "complete.obs")
corrplot(
cor_matrix,
method = "color",
# type = "lower",
tl.col = "black",
tl.srt = 45,
tl.cex = 0.8,
number.cex = 0.7,
addCoef.col = "black",
diag = FALSE,
col = colorRampPalette(c("blue", "white", "red"))(200),
mar = c(0,0,1,0),
title = "Correlation Heatmap of Delay Variables"
)
The correlation heatmap reveals a very strong positive correlation between departure and arrival delays, as expected. There are also moderate correlations between carrier, NAS, and late aircraft delay variables, indicating that these causes often co-occur. Most other variables show weak or negligible correlations, suggesting that delay causes are relatively independent except for a few key relationships.
4.3. Time Dimension Analysis
4.3.1 Average Departure Delay by Hour
flights %>%
group_by(DEP_HOUR) %>%
summarise(avg_delay = mean(DEP_DELAY, na.rm = TRUE)) %>%
ggplot(aes(x = DEP_HOUR, y = avg_delay)) +
geom_line(color = "steelblue", size = 1) +
geom_point(color = "steelblue", size = 2) +
labs(title = "Average Departure Delay by Hour",
x = "Hour of Day",
y = "Average Departure Delay (minutes)") +
theme_minimal() +
scale_x_continuous(breaks = 0:23)
The line plot indicates that average departure delays are lowest in the early morning hours, but increase steadily throughout the day, peaking in the late evening. There is a noticeable spike around 2-3 AM, which may be due to a small number of flights or specific operational patterns. The general upward trend suggests that delays accumulate as the day progresses, likely due to cascading effects from earlier disruptions.
4.3.2 Violin Plot of Departure Delays by Day of Week
ggplot(flights, aes(x = WEEKDAY, y = DEP_DELAY)) +
geom_violin(fill = "steelblue", alpha = 0.7) +
labs(title = "Departure Delays by Day of Week",
x = "Day of Week",
y = "Departure Delay (minutes)") +
theme_minimal() +
scale_y_continuous(limits = c(-30, 120))
The violin plot shows that the distribution of departure delays is remarkably consistent across all days of the week. Each day exhibits a concentration of short delays and a long tail of extreme values, with no significant differences between weekdays and weekends. This suggests that day-of-week is not a major factor influencing departure delay patterns.
4.3.3 Average Departure Delay by Month
flights %>%
group_by(MONTH) %>%
summarise(avg_delay = mean(DEP_DELAY, na.rm = TRUE)) %>%
ggplot(aes(x = MONTH, y = avg_delay)) +
geom_bar(stat = "identity", fill = "steelblue", alpha = 0.7) +
labs(title = "Average Departure Delay by Month",
x = "Month",
y = "Average Departure Delay (minutes)") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
The bar chart reveals clear seasonal variation in average departure delays. Delays are highest in June and July, likely reflecting increased travel demand and potential weather disruptions during the summer months. There is also a secondary peak in December, possibly due to holiday travel. The lowest delays occur in September and November, indicating more stable operational conditions during these months.
4.4. Flight Status Analysis
4.4.1 Cancellation Rate by Airline
flights %>%
group_by(AIRLINE_CODE) %>%
summarise(
total_flights = n(),
cancelled_flights = sum(CANCELLED == 1),
cancellation_rate = cancelled_flights / total_flights
) %>%
ggplot(aes(x = reorder(AIRLINE_CODE, cancellation_rate), y = cancellation_rate)) +
geom_bar(stat = "identity", fill = "steelblue", alpha = 0.7) +
labs(title = "Cancellation Rate by Airline",
x = "Airline Code",
y = "Cancellation Rate") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
scale_y_continuous(labels = percent)
This bar chart shows substantial variation in cancellation rates among airlines. While most airlines maintain cancellation rates below 3%, a few—such as G4 and EV—exhibit much higher rates, exceeding 4%. These differences may reflect variations in operational resilience, route networks, or responses to adverse conditions.
4.4.2 Actual vs. Scheduled Departure Time
ggplot(flights, aes(x = CRS_DEP_TIME, y = DEP_TIME)) +
geom_point(alpha = 0.1, color = "steelblue") +
geom_abline(intercept = 0, slope = 1, color = "red", linetype = "dashed") +
labs(title = "Actual vs. Scheduled Departure Time",
x = "Scheduled Departure Time",
y = "Actual Departure Time") +
theme_minimal()
The scatter plot illustrates that while many flights depart close to their scheduled times (along the diagonal), there is a significant spread above the line, indicating late departures. The vertical and horizontal banding patterns reflect the use of scheduled departure times at regular intervals, and the presence of many points above the diagonal highlights the prevalence of delays.
4.4.3 Diverted Flights by Origin Airport
flights %>%
filter(DIVERTED == 1) %>%
group_by(ORIGIN) %>%
summarise(diverted_count = n()) %>%
arrange(desc(diverted_count)) %>%
head(20) %>%
ggplot(aes(x = reorder(ORIGIN, diverted_count), y = diverted_count)) +
geom_bar(stat = "identity", fill = "steelblue", alpha = 0.7) +
labs(title = "Top 20 Origin Airports by Diverted Flights",
x = "Origin Airport",
y = "Number of Diverted Flights") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
coord_flip()
This bar chart identifies the airports with the highest number of diverted flights, with ORD, ATL, and DFW leading by a significant margin. These airports are major hubs and may be more susceptible to diversions due to high traffic volumes, weather variability, or airspace constraints.
4.5. Advanced Analysis
4.5.1 Pair Plot of Delay-Related Variables
delay_vars <- flights %>%
select(DEP_DELAY, ARR_DELAY, TAXI_OUT, TAXI_IN, AIR_TIME) %>%
sample_n(1000)
pairs(delay_vars,
main = "Pair Plot of Delay-Related Variables",
pch = 19,
col = alpha("steelblue", 0.3))
The pair plot provides a comprehensive view of the relationships between key delay-related variables. The strong linear association between departure and arrival delays is evident, as is the positive relationship between taxi-out time and both types of delays. Other variable pairs show weaker or no clear relationships, indicating that not all operational metrics are closely linked.
4.5.2 Average Departure Delay by Flight Distance Bins
flights %>%
mutate(distance_bin = cut(DISTANCE,
breaks = c(0, 500, 1000, 1500, 2000, Inf),
labels = c("0-500", "501-1000", "1001-1500",
"1501-2000", "2000+"))) %>%
group_by(distance_bin) %>%
summarise(avg_delay = mean(DEP_DELAY, na.rm = TRUE)) %>%
ggplot(aes(x = distance_bin, y = avg_delay)) +
geom_bar(stat = "identity", fill = "steelblue", alpha = 0.7) +
labs(title = "Average Departure Delay by Flight Distance Bin",
x = "Flight Distance Bin (miles)",
y = "Average Departure Delay (minutes)") +
theme_minimal()
The bar chart shows that average departure delays increase with flight distance up to the 1001-1500 mile range, after which they plateau or slightly decrease. Short-haul flights (0-500 miles) experience the lowest average delays, while medium-haul flights (1001-2000 miles) are most affected. This pattern may reflect differences in scheduling buffers or operational complexity across distance categories.
4.5.3 Delay Causes Over Time
delay_causes_by_month <- flights %>%
group_by(MONTH) %>%
summarise(
Carrier = mean(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = mean(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = mean(DELAY_DUE_NAS, na.rm = TRUE),
Security = mean(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = mean(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
pivot_longer(-MONTH, names_to = "Cause", values_to = "AvgDelay")
ggplot(delay_causes_by_month, aes(x = MONTH, y = AvgDelay, fill = Cause)) +
geom_col(position = "dodge") +
labs(title = "Delay Causes Over Time",
x = "Month",
y = "Average Delay Minutes",
fill = "Delay Cause") +
theme_minimal() +
scale_fill_viridis_d() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
The grouped bar chart demonstrates that carrier and late aircraft delays are consistently the largest contributors to total delay minutes throughout the year. Weather-related delays show a modest increase in the winter months, while NAS and security delays remain relatively stable. This seasonal pattern highlights the persistent impact of airline operations on delays, with weather playing a secondary but notable role during certain periods.
4.5.4 Top 10 Routes by Average Departure Delay
flights %>%
group_by(ORIGIN, DEST) %>%
summarise(avg_delay = mean(DEP_DELAY, na.rm = TRUE),
route = paste(ORIGIN, "-", DEST)) %>%
arrange(desc(avg_delay)) %>%
head(10) %>%
ggplot(aes(x = reorder(route, avg_delay), y = avg_delay)) +
geom_bar(stat = "identity", fill = "steelblue", alpha = 0.7) +
labs(title = "Top 10 Routes by Average Departure Delay",
x = "Route",
y = "Average Departure Delay (minutes)") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
coord_flip()
This bar chart highlights a small number of routes with exceptionally high average departure delays, with SFB-GFK, FCA-LGA, and FOD-DEN standing out as extreme cases. These routes may be subject to unique operational challenges or recurring disruptions, making them prime candidates for targeted investigation and intervention.
#5. Modeling
5.1 Predicting flight delay based on linear regression model
# Read dataset
flights <- read.csv("flights_cleaned002.csv")
flights <- flights[1:200000, ] # due to computer performance reasons, the first 200,000 data are selected here.
# Select relevant features and handle missing values
model_data <- flights %>%
select(
DEP_DELAY, # Target variable (departure delay in minutes)
AIRLINE_CODE, # Airline carrier code
ORIGIN, # Origin airport code
DEST, # Destination airport code
CRS_DEP_TIME, # Scheduled departure time
DISTANCE, # Flight distance
CRS_ELAPSED_TIME # Scheduled flight duration
) %>%
na.omit() %>% # Remove rows with missing values
mutate(
# Convert categorical variables to factors
AIRLINE_CODE = as.factor(AIRLINE_CODE),
ORIGIN = as.factor(ORIGIN),
DEST = as.factor(DEST),
# Create time-of-day features
DEP_HOUR = floor(CRS_DEP_TIME / 100),
DEP_PERIOD = case_when(
DEP_HOUR >= 5 & DEP_HOUR < 12 ~ "Morning",
DEP_HOUR >= 12 & DEP_HOUR < 17 ~ "Afternoon",
DEP_HOUR >= 17 & DEP_HOUR < 22 ~ "Evening",
TRUE ~ "Night"
)
)
# Split data into training (70%) and testing (30%) sets
set.seed(123)
train_index <- createDataPartition(model_data$DEP_DELAY, p = 0.7, list = FALSE)
train_data <- model_data[train_index, ]
test_data <- model_data[-train_index, ]Regression Model Building We build a linear regression model to predict DEP_DELAY using selected features.
# Train linear regression model
model <- train(
DEP_DELAY ~ AIRLINE_CODE + ORIGIN + DEST +
DEP_PERIOD + DISTANCE + CRS_ELAPSED_TIME,
data = train_data,
method = "lm",
trControl = trainControl(method = "cv", number = 5) # 5-fold cross-validation
)Predictions and Residuals Generate predictions and compute residuals for diagnostic purposes.
# Generate predictions on test set
test_data$pred_delay <- predict(model, newdata = test_data)
# Calculate residuals
test_data$residual <- test_data$DEP_DELAY - test_data$pred_delay
# Create results dataframe
results <- data.frame(
Actual = test_data$DEP_DELAY,
Predicted = test_data$pred_delay,
Residual = test_data$residual,
Airline = test_data$AIRLINE_CODE,
Origin = test_data$ORIGIN
)
# Display first 10 predictions
head(results, 10)## Actual Predicted Residual Airline Origin
## 1 -6 12.404734 -18.404734 DL MSP
## 2 147 8.414697 138.585303 WN DAL
## 3 -9 9.718266 -18.718266 AS SEA
## 4 -5 1.638173 -6.638173 DL BDL
## 5 -4 16.638700 -20.638700 UA SRQ
## 6 13 10.887057 2.112943 NK FLL
## 7 -7 9.844314 -16.844314 EV TUL
## 8 -5 9.762425 -14.762425 AA DFW
## 9 30 13.557501 16.442499 OO IAH
## 10 -4 15.709433 -19.709433 WN BNA# Plot residuals vs. fitted values
ggplot(test_data, aes(x = pred_delay, y = residual)) +
geom_point(alpha = 0.5) +
geom_hline(yintercept = 0, linetype = "dashed", color = "red") +
theme_minimal() +
labs(title = "Residuals vs. Fitted Values", x = "Predicted Delay (minutes)", y = "Residuals")
Model Evaluation (RMSE) Evaluate the model using Root Mean Squared Error (RMSE)
# Calculate RMSE
rmse <- sqrt(mean(test_data$residual^2))
# Print evaluation metrics
cat(sprintf("Model Performance Metrics:\n----------------------------------\nRoot Mean Squared Error (RMSE): %.2f minutes\nR-squared (Training): %.4f\n", rmse, model$results$Rsquared))## Model Performance Metrics:
## ----------------------------------
## Root Mean Squared Error (RMSE): 50.91 minutes
## R-squared (Training): 0.0081The linear regression model’s performance is poor, with an RMSE of 299.17 minutes indicating large prediction errors and an R-squared of 0.1682 showing that it explains only 16.82% of the variance in flight departure delays. This model is inadequate for practical use due to its inability to capture the complex, non-linear nature of flight delays. Switching to a more advanced model like XGBoost, incorporating additional features, and using a larger dataset are recommended to improve predictive accuracy.
5.2 Predicting flight delay based on XGBoost model
library(dplyr)
library(caret)
library(ggplot2)
library(xgboost)
flights <- read.csv("flights_cleaned002.csv")
flights <- flights[1:200000, ]
# Select relevant features and handle missing values
model_data <- flights %>%
select(
DEP_DELAY, # Target variable (departure delay in minutes)
AIRLINE_CODE, # Airline carrier code
ORIGIN, # Origin airport code
DEST, # Destination airport code
CRS_DEP_TIME, # Scheduled departure time
DISTANCE, # Flight distance
CRS_ELAPSED_TIME, # Scheduled flight duration
DELAY_DUE_CARRIER, # Carrier-related delay
DELAY_DUE_WEATHER, # Weather-related delay
DELAY_DUE_NAS, # National Airspace System delay
DELAY_DUE_SECURITY, # Security-related delay
DELAY_DUE_LATE_AIRCRAFT # Late aircraft delay
) %>%
mutate(
# Convert categorical variables to factors
AIRLINE_CODE = as.factor(AIRLINE_CODE),
ORIGIN = as.factor(ORIGIN),
DEST = as.factor(DEST),
# Create time-based features
DEP_HOUR = floor(CRS_DEP_TIME / 100),
DEP_PERIOD = case_when(
DEP_HOUR >= 5 & DEP_HOUR < 12 ~ "Morning",
DEP_HOUR >= 12 & DEP_HOUR < 17 ~ "Afternoon",
DEP_HOUR >= 17 & DEP_HOUR < 22 ~ "Evening",
TRUE ~ "Night"
),
DEP_PERIOD = as.factor(DEP_PERIOD),
# Simplify ORIGIN and DEST into top 10 airports and 'Other'
ORIGIN_GROUP = if_else(ORIGIN %in% c("ATL", "DFW", "DEN", "ORD", "LAX",
"JFK", "SFO", "SEA", "MCO", "EWR"),
ORIGIN, "Other"),
DEST_GROUP = if_else(DEST %in% c("ATL", "DFW", "DEN", "ORD", "LAX",
"JFK", "SFO", "SEA", "MCO", "EWR"),
DEST, "Other"),
ORIGIN_GROUP = as.factor(ORIGIN_GROUP),
DEST_GROUP = as.factor(DEST_GROUP),
# Handle negative DEP_DELAY for log-transformation
DEP_DELAY = pmax(DEP_DELAY, 0), # Cap negative delays at 0
LOG_DEP_DELAY = log(DEP_DELAY + 1) # Log-transform
) %>%
# Impute missing values for numeric predictors
mutate(
DISTANCE = replace(DISTANCE, is.na(DISTANCE), mean(DISTANCE, na.rm = TRUE)),
CRS_ELAPSED_TIME = replace(CRS_ELAPSED_TIME, is.na(CRS_ELAPSED_TIME),
mean(CRS_ELAPSED_TIME, na.rm = TRUE)),
DELAY_DUE_CARRIER = replace(DELAY_DUE_CARRIER, is.na(DELAY_DUE_CARRIER), 0),
DELAY_DUE_WEATHER = replace(DELAY_DUE_WEATHER, is.na(DELAY_DUE_WEATHER), 0),
DELAY_DUE_NAS = replace(DELAY_DUE_NAS, is.na(DELAY_DUE_NAS), 0),
DELAY_DUE_SECURITY = replace(DELAY_DUE_SECURITY, is.na(DELAY_DUE_SECURITY), 0),
DELAY_DUE_LATE_AIRCRAFT = replace(DELAY_DUE_LATE_AIRCRAFT,
is.na(DELAY_DUE_LATE_AIRCRAFT), 0)
) %>%
# Filter out rows with NA, NaN, or Inf
filter(
!is.na(DEP_DELAY) & !is.na(DEP_HOUR) &
!is.na(LOG_DEP_DELAY) & !is.infinite(LOG_DEP_DELAY) &
!is.na(DISTANCE) & !is.na(CRS_ELAPSED_TIME) &
!is.na(AIRLINE_CODE) & !is.na(ORIGIN_GROUP) & !is.na(DEST_GROUP) &
DEP_DELAY <= 1440
)
# Split data into training (70%) and testing (30%) sets
set.seed(123)
train_index <- createDataPartition(model_data$LOG_DEP_DELAY, p = 0.7, list = FALSE)
train_data <- model_data[train_index, ]
test_data <- model_data[-train_index, ]This code block loads the flight dataset (flights_cleaned002.csv) and selects the first 200,000 rows for computational efficiency.Categorical variables are converted to factors, and new features (DEP_HOUR, DEP_PERIOD, ORIGIN_GROUP, DEST_GROUP) are created. DEP_DELAY is log-transformed (LOG_DEP_DELAY) after capping negative values at 0 to handle non-positive values. Missing values in numeric predictors are imputed with their means, and rows with invalid values (NA, NaN, Inf) or extreme delays (>1440 minutes) are filtered out. The data is split into 70% training and 30% testing sets.
# Define model formula for XGBoost
model_formula_xgb <- LOG_DEP_DELAY ~ AIRLINE_CODE + ORIGIN_GROUP + DEST_GROUP +
DEP_HOUR + DEP_PERIOD + DISTANCE + CRS_ELAPSED_TIME +
DELAY_DUE_CARRIER + DELAY_DUE_WEATHER +
DELAY_DUE_NAS + DELAY_DUE_SECURITY +
DELAY_DUE_LATE_AIRCRAFT
# Train XGBoost model
xgb_model <- train(
model_formula_xgb,
data = train_data,
method = "xgbTree",
trControl = trainControl(method = "cv", number = 5, verboseIter = FALSE),
tuneGrid = expand.grid(
nrounds = c(50, 100),
max_depth = c(3, 6),
eta = c(0.1, 0.3),
gamma = 0,
colsample_bytree = 0.8,
min_child_weight = 1,
subsample = 0.8
)
)## [16:23:57] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:05] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:09] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:16] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:20] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:27] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:32] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:40] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:44] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:52] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:24:57] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:05] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:09] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:16] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:21] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:28] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:32] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:40] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:44] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.
## [16:25:52] WARNING: src/c_api/c_api.cc:935: `ntree_limit` is deprecated, use `iteration_range` instead.This section defines a formula for the XGBoost model to predict LOG_DEP_DELAY using multiple predictors. The train function from the caret package is used to train an XGBoost model (xgbTree) with 5-fold cross-validation. A tuning grid is specified to test combinations of hyperparameters (nrounds, max_depth, eta, etc.) to optimize model performance.
# XGBoost predictions
test_data$pred_log_delay_xgb <- predict(xgb_model, newdata = test_data)
test_data$pred_delay_xgb <- exp(test_data$pred_log_delay_xgb) - 1
test_data$residual_xgb <- test_data$DEP_DELAY - test_data$pred_delay_xgb
# Create results dataframe
results <- data.frame(
Actual = test_data$DEP_DELAY,
Predicted_XGB = test_data$pred_delay_xgb,
Residual_XGB = test_data$residual_xgb,
Airline = test_data$AIRLINE_CODE,
Origin = test_data$ORIGIN_GROUP
)
# Display first 10 predictions
knitr::kable(head(results, 10))| Actual | Predicted_XGB | Residual_XGB | Airline | Origin |
|---|---|---|---|---|
| 0 | 0.6239434 | -0.6239434 | UA | Other |
| 6 | 0.4873009 | 5.5126991 | UA | DEN |
| 0 | 1.9917278 | -1.9917278 | DL | Other |
| 0 | 1.0342706 | -1.0342706 | NK | MCO |
| 0 | 0.3550785 | -0.3550785 | AA | Other |
| 0 | 0.9632622 | -0.9632622 | AS | SEA |
| 6 | 0.5452601 | 5.4547399 | WN | Other |
| 0 | 0.1951116 | -0.1951116 | DL | Other |
| 0 | 0.5618880 | -0.5618880 | UA | ORD |
| 0 | 0.4873161 | -0.4873161 | G4 | Other |
# Plot residuals vs. fitted values (XGBoost)
ggplot(test_data, aes(x = pred_delay_xgb, y = residual_xgb)) +
geom_point(alpha = 0.5) +
geom_hline(yintercept = 0, linetype = "dashed", color = "red") +
theme_minimal() +
labs(title = "XGBoost: Residuals vs. Fitted Values",
x = "Predicted Delay (minutes)", y = "Residuals")
This code block generates predictions using the trained XGBoost model on
the test set, storing the log-transformed predictions in
pred_log_delay_xgb. These are exponentiated and adjusted
(exp(…) - 1) to obtain predictions in the original scale
(pred_delay_xgb). Residuals are calculated as the difference
between actual (DEP_DELAY) and predicted (pred_delay_xgb) values. A
results data frame is created to store actual, predicted, and residual
values along with AIRLINE_CODE and ORIGIN_GROUP
# XGBoost evaluation
rmse_xgb <- sqrt(mean(test_data$residual_xgb^2, na.rm = TRUE))
ss_total <- sum((test_data$DEP_DELAY - mean(test_data$DEP_DELAY, na.rm = TRUE))^2, na.rm = TRUE)
ss_residual_xgb <- sum(test_data$residual_xgb^2, na.rm = TRUE)
r_squared_xgb <- 1 - ss_residual_xgb / ss_total
mae_xgb <- mean(abs(test_data$residual_xgb), na.rm = TRUE)
# Print evaluation metrics
cat(sprintf("Model Performance Metrics:\nXGBoost:\n----------------------------------\nRoot Mean Squared Error (RMSE): %.2f minutes\nR-squared: %.4f\nMean Absolute Error (MAE): %.2f minutes\n", rmse_xgb, r_squared_xgb, mae_xgb))## Model Performance Metrics:
## XGBoost:
## ----------------------------------
## Root Mean Squared Error (RMSE): 13.13 minutes
## R-squared: 0.9291
## Mean Absolute Error (MAE): 3.86 minutesThe XGBoost model demonstrates strong performance with an R-squared of 0.9095, indicating it explains nearly 91% of the variance in flight departure delays. The RMSE of 15.26 minutes suggests moderate predictive accuracy, suitable for most airline planning purposes but potentially improvable for high-precision needs.
# 6.Delay Reason Analysis
df <- read.csv("flights_cleaned002.csv", check.names = TRUE)6.1. Overview of Delay Cause Variables
Before proceeding to visualizations, we performed a basic descriptive
analysis of the five delay cause variables:
DELAY_DUE_CARRIER, DELAY_DUE_WEATHER,
DELAY_DUE_NAS, DELAY_DUE_SECURITY, and
DELAY_DUE_LATE_AIRCRAFT. These variables record the delay
minutes attributed to each cause for each flight. The analysis includes
summary statistics, mean delay minutes, and missing value counts,
providing an overview of each delay type’s scale and data quality.
# Select delay cause columns
delay_causes <- df[, c("DELAY_DUE_CARRIER", "DELAY_DUE_WEATHER",
"DELAY_DUE_NAS", "DELAY_DUE_SECURITY",
"DELAY_DUE_LATE_AIRCRAFT")]
# Summary statistics
print("=== Summary Statistics ===")## [1] "=== Summary Statistics ==="print(summary(delay_causes))## DELAY_DUE_CARRIER DELAY_DUE_WEATHER DELAY_DUE_NAS DELAY_DUE_SECURITY
## Min. : 0.0 Min. : 0.0 Min. : 0.0 Min. : 0.0
## 1st Qu.: 0.0 1st Qu.: 0.0 1st Qu.: 0.0 1st Qu.: 0.0
## Median : 4.0 Median : 0.0 Median : 0.0 Median : 0.0
## Mean : 24.6 Mean : 3.9 Mean : 13.2 Mean : 0.1
## 3rd Qu.: 23.0 3rd Qu.: 0.0 3rd Qu.: 17.0 3rd Qu.: 0.0
## Max. :2685.0 Max. :1398.0 Max. :1468.0 Max. :377.0
## NA's :821483 NA's :821483 NA's :821483 NA's :821483
## DELAY_DUE_LATE_AIRCRAFT
## Min. : 0.0
## 1st Qu.: 0.0
## Median : 0.0
## Mean : 25.4
## 3rd Qu.: 30.0
## Max. :2096.0
## NA's :821483# Average delay per cause
print("=== Mean Delay Minutes per Cause ===")## [1] "=== Mean Delay Minutes per Cause ==="print(colMeans(delay_causes, na.rm = TRUE))## DELAY_DUE_CARRIER DELAY_DUE_WEATHER DELAY_DUE_NAS
## 24.5500373 3.9218338 13.1702975
## DELAY_DUE_SECURITY DELAY_DUE_LATE_AIRCRAFT
## 0.1487701 25.4001524# Missing value counts
print("=== Missing Values per Cause ===")## [1] "=== Missing Values per Cause ==="print(sapply(delay_causes, function(x) sum(is.na(x))))## DELAY_DUE_CARRIER DELAY_DUE_WEATHER DELAY_DUE_NAS
## 821483 821483 821483
## DELAY_DUE_SECURITY DELAY_DUE_LATE_AIRCRAFT
## 821483 821483The summary statistics reveal that the most significant sources of
flight delays are DELAY_DUE_CARRIER and
DELAY_DUE_LATE_AIRCRAFT, with average delay minutes of
approximately 24.6 and 25.4, respectively. These two causes also have
the largest maximum recorded delays, exceeding 2000 minutes.
DELAY_DUE_NAS shows a moderate average delay of 13.2
minutes, while DELAY_DUE_WEATHER contributes a relatively
small amount at around 3.9 minutes. DELAY_DUE_SECURITY has
a negligible mean value, indicating it is rarely a substantial
factor.
All five variables contain a substantial number of missing values (821,483), which likely corresponds to flights that were not affected by that specific delay cause. This pattern is consistent with operational expectations, as not all flights encounter delays, and among those that do, only certain causes may be relevant.
6.2.1 Proportional Delay Structure by Airline
To better understand the internal structure of flight delays across airlines, we plotted a percentage-stacked bar chart showing the relative contributions of each delay cause for every airline. Unlike absolute delay minutes, this chart normalizes the total delays per airline, allowing for direct comparison of delay patterns regardless of airline size.
The visualization reveals that some airlines are predominantly affected by specific causes. For example, late aircraft and carrier-related issues appear to dominate the delay structure for most airlines, whereas weather and security-related delays consistently account for smaller proportions. This highlights operational patterns and potential inefficiencies that are more structural rather than volume-driven.
Such proportional insights are useful for identifying systemic issues within specific carriers and for prioritizing targeted interventions.
library(dplyr)
library(tidyr)
library(ggplot2)
# Step 1: Aggregate delay minutes by airline code
airline_delay_raw <- df %>%
group_by(AIRLINE) %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
)
# Step 2: Pivot to long format
airline_delay_long <- airline_delay_raw %>%
pivot_longer(cols = Carrier:Late_Aircraft, names_to = "Cause", values_to = "Minutes")
# Step 3: Calculate percentage within each airline
airline_delay_pct <- airline_delay_long %>%
group_by(AIRLINE) %>%
mutate(Percentage = Minutes / sum(Minutes))
# Step 4: Draw the percentage stacked bar chart
ggplot(airline_delay_pct, aes(x = reorder(AIRLINE, -Percentage), y = Percentage, fill = Cause)) +
geom_bar(stat = "identity", position = "stack") +
scale_y_continuous(labels = scales::percent_format()) +
labs(title = "Percentage of Delay Causes by Airline",
x = "Airline",
y = "Proportion of Total Delay Minutes",
fill = "Delay Cause") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
The percentage stacked bar chart provides a normalized comparison of
delay causes across different airlines. Despite variations in the total
number of delays among airlines, this visualization allows for clearer
identification of dominant delay types by relative contribution.
It is evident that carrier-related delays constitute the majority of delay minutes for most airlines, particularly for regional carriers such as Endeavor Air and PSA Airlines, where over 60% of delays are attributed to airline operations. Late aircraft delays also represent a substantial share across many carriers, indicating systemic scheduling or turnaround issues.
In contrast, weather- and security-related delays consistently account for the smallest proportions, reinforcing the view that operational and logistical inefficiencies, rather than uncontrollable external factors, are the dominant causes of delay.
Overall, this chart emphasizes the need for airline-specific strategies focused on operational performance and aircraft turnaround optimization, rather than solely relying on broader systemic improvements.
6.2.2 Delay Causes by Origin Airport
To better understand how flight delay causes vary by airport, we created a heatmap that visualizes the total delay minutes by cause for the top 15 origin airports. Each cell reflects the intensity of delays for a specific cause at a given airport, allowing easy comparison across both dimensions. This layout provides a more intuitive view than bubble charts when comparing categorical combinations.
library(ggplot2)
library(dplyr)
# Step 1: Aggregate delay data by airport and cause
top_airports <- df %>%
group_by(ORIGIN) %>%
summarise(TotalDelay = sum(DELAY_DUE_CARRIER + DELAY_DUE_WEATHER +
DELAY_DUE_NAS + DELAY_DUE_SECURITY +
DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)) %>%
arrange(desc(TotalDelay)) %>%
slice_head(n = 15) %>%
pull(ORIGIN)
airport_heatmap <- df %>%
filter(ORIGIN %in% top_airports) %>%
group_by(ORIGIN) %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
tidyr::pivot_longer(cols = -ORIGIN, names_to = "Cause", values_to = "Total_Delay")
# Step 2: Draw heatmap
ggplot(airport_heatmap, aes(x = Cause, y = ORIGIN, fill = Total_Delay)) +
geom_tile(color = "white") +
scale_fill_gradient(low = "lightyellow", high = "red") +
labs(title = "Heatmap of Delay Causes by Origin Airport",
x = "Delay Cause",
y = "Origin Airport",
fill = "Total Delay (mins)") +
theme_minimal() +
theme(axis.text.x = element_text(angle = 45, hjust = 1))
The heatmap above presents a clear overview of how different delay causes vary across the top 15 origin airports. The intensity of each cell indicates the total delay minutes attributed to a specific cause at each airport.
It is evident that late aircraft delays are the most dominant cause of delays across nearly all airports, particularly at DFW (Dallas/Fort Worth) and EWR (Newark), where the red shades indicate over 250,000 minutes of delay. Carrier-related delays also appear prominent at ATL, DEN, and ORD, suggesting possible airline-specific inefficiencies at these hubs.
In contrast, security-related delays are minimal across all airports, while weather-related delays show relatively uniform but moderate impact, with slightly higher values at ORD and LGA, possibly due to frequent weather disruptions in those regions.
This visualization helps identify where mitigation efforts should be focused. Airports with consistently high delays from specific causes—especially late aircraft—may benefit from improved turnaround scheduling or operational buffers.
6.2.3 Delay Causes by Hour of Day
To better understand the variation of delay causes throughout the day, we analyze the total delay minutes for each cause across different hours using a stacked line plot. This visualization allows us to observe when specific delay types become more prominent, enabling better resource planning and targeted mitigation during peak periods.
library(dplyr)
library(tidyr)
library(ggplot2)
# Step 1: Ensure hour column is clean and numeric
df$CRSDepHour <- as.integer(df$CRSDepHour)
# Step 2: Aggregate delay minutes by hour and cause
hourly_delay <- df %>%
filter(!is.na(CRSDepHour)) %>%
group_by(CRSDepHour) %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
pivot_longer(cols = -CRSDepHour, names_to = "Cause", values_to = "Total_Delay")
# Step 3: Stacked Line Plot using geom_area
ggplot(hourly_delay, aes(x = CRSDepHour, y = Total_Delay, fill = Cause)) +
geom_area(alpha = 0.9, position = "stack") +
scale_x_continuous(breaks = 0:23) +
labs(
title = "Hourly Distribution of Delay Causes (Stacked Line Plot)",
x = "Hour of Day",
y = "Total Delay Minutes",
fill = "Delay Cause"
) +
theme_minimal()
The stacked line plot illustrates the hourly distribution of flight
delays segmented by cause. Delays tend to rise sharply starting around 6
AM, reaching a peak between 4 PM and 6 PM. During these peak hours,
Carrier and Late_Aircraft delays dominate the
total delay minutes, suggesting significant congestion and turnaround
inefficiencies in the late afternoon.
Notably, NAS (National Airspace System) and
Security delays also show an increase during midday, while
Weather delays appear relatively consistent throughout the
day, with a slight uptick during late afternoon hours. Early morning
hours (midnight to 5 AM) show minimal delay activity across all
causes.
These patterns highlight the importance of time-based delay mitigation efforts, especially focusing on carrier operations and aircraft readiness in the afternoon, and monitoring airspace congestion during midday.
6.2.4 Delay Causes by Day of Week
To examine how flight delay causes vary across different days of the
week, we analyze the total delay minutes for each cause grouped by
weekday (Weekday). This allows us to identify whether
specific delay types, such as carrier or weather-related delays, are
more prevalent on certain days. The resulting stacked bar chart enables
an intuitive comparison across all seven days, providing insights into
operational patterns and informing better scheduling or staffing
decisions at airports.
library(dplyr)
library(tidyr)
library(ggplot2)
# Step 1: Ensure Weekday column is present and ordered from Monday to Sunday
df$Weekday <- factor(df$Weekday,
levels = c("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"))
# Step 2: Aggregate total delay minutes for each weekday and cause
weekday_delay <- df %>%
filter(!is.na(Weekday)) %>%
group_by(Weekday) %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
pivot_longer(cols = -Weekday, names_to = "Cause", values_to = "Total_Delay")
# Step 3: Create stacked bar chart
ggplot(weekday_delay, aes(x = Weekday, y = Total_Delay, fill = Cause)) +
geom_bar(stat = "identity") +
labs(title = "Delay Causes by Day of Week",
x = "Weekday",
y = "Total Delay Minutes",
fill = "Delay Cause") +
theme_minimal()
The stacked bar chart illustrates how flight delay causes are distributed across different days of the week. Notably, Thursday and Friday exhibit the highest total delay minutes, followed closely by Monday and Sunday. These peaks are primarily attributed to carrier-related and late aircraft delays, suggesting that operational pressure may intensify toward the end of the workweek and at the start of the travel-heavy weekend.
In contrast, Tuesday and Wednesday show significantly lower total delays, indicating relatively smoother airline operations during midweek. Additionally, NAS (National Airspace System) delays remain consistently present throughout the week, while security and weather-related delays contribute the least and exhibit minimal day-to-day variation.
These findings imply that airlines and airport authorities might benefit from allocating additional resources and contingency plans specifically on high-traffic days like Thursday, Friday, and Sunday to mitigate delay risks and improve service reliability.
6.2.5 Delay Causes by Holiday Status
To investigate whether holidays influence flight delays, we compare the total delay minutes by cause for flights occurring on holidays versus non-holidays. By using a stacked bar chart, we can observe how each delay cause contributes to the overall delay under both conditions. This comparison may reveal whether certain causes, such as weather or late aircraft, are more prominent during holidays due to increased travel volume or operational constraints.
# Load required library for plotting
library(ggplot2)
# Step 1: Group by holiday status and summarize total delay minutes
holiday_delay <- df %>%
group_by(IsHoliday) %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
pivot_longer(cols = -IsHoliday, names_to = "Cause", values_to = "Total_Delay")
# Step 2: Create a stacked bar chart to compare holiday vs non-holiday
ggplot(holiday_delay, aes(x = IsHoliday, y = Total_Delay, fill = Cause)) +
geom_bar(stat = "identity") +
labs(title = "Delay Causes by Holiday Status",
x = "Is Holiday",
y = "Total Delay Minutes",
fill = "Delay Cause") +
theme_minimal()
The stacked bar chart comparing delay causes by holiday status reveals a
notable difference in overall delay volumes. Flights on non-holidays
(FALSE) account for significantly more total delay minutes across all
causes, which is expected given that most flights occur on regular days.
However, even during holidays (TRUE), substantial delays are still
recorded.
Late aircraft and carrier-related issues remain the dominant delay causes in both scenarios. Interestingly, the relative proportion of these causes does not shift drastically between holiday and non-holiday flights. This suggests that the mechanisms causing delays are fairly consistent year-round, although operational intensity may increase during holidays.
Overall, while holidays do not introduce new dominant causes, the volume reduction highlights the potential for reduced congestion or improved management during these peak seasons, possibly due to advance planning and resource allocation.
6.2.6 Delay Causes by Flight Distance Bin
To assess whether flight distance influences the nature and severity
of delays, we analyze total delay minutes by cause across different
flight distance bins. The Distance_Bin variable categorizes
flights into ranges (e.g., 0–500 miles, 501–1000 miles), allowing us to
compare short-haul and long-haul routes. This comparison provides
insight into whether longer flights are more susceptible to specific
delay causes.
# Load required libraries
library(dplyr)
library(ggplot2)
library(tidyr)
# Step 1: Create a new distance bin column
df$DISTANCE_BIN <- cut(df$DISTANCE,
breaks = c(0, 500, 1000, 1500, 2000, Inf),
labels = c("0–500", "501–1000", "1001–1500", "1501–2000", "2000+"),
right = FALSE)
# Step 2: Aggregate total delay minutes for each bin and cause
distance_delay <- df %>%
group_by(DISTANCE_BIN) %>%
summarise(
Carrier = sum(DELAY_DUE_CARRIER, na.rm = TRUE),
Weather = sum(DELAY_DUE_WEATHER, na.rm = TRUE),
NAS = sum(DELAY_DUE_NAS, na.rm = TRUE),
Security = sum(DELAY_DUE_SECURITY, na.rm = TRUE),
Late_Aircraft = sum(DELAY_DUE_LATE_AIRCRAFT, na.rm = TRUE)
) %>%
pivot_longer(cols = -DISTANCE_BIN, names_to = "Cause", values_to = "Total_Delay")
# Step 3: Plot stacked bar chart
ggplot(distance_delay, aes(x = DISTANCE_BIN, y = Total_Delay, fill = Cause)) +
geom_bar(stat = "identity") +
labs(title = "Delay Causes by Flight Distance Bin",
x = "Flight Distance Bin",
y = "Total Delay Minutes",
fill = "Delay Cause") +
theme_minimal()
The stacked bar chart reveals that short- to medium-distance flights
(0–1000 miles) account for the majority of total delay minutes. In
particular, the 501–1000 miles bin shows the highest cumulative delays
across all causes, followed closely by the 0–500 miles bin. Carrier and
late aircraft delays dominate within these ranges, indicating that
operational bottlenecks and aircraft turnaround issues are more common
in high-frequency, short-haul operations.
As flight distances increase beyond 1500 miles, the total delay minutes decrease significantly across all categories. This suggests that long-haul flights tend to be more efficiently scheduled and less affected by the same operational disruptions, potentially due to more buffer time built into their schedules or lower flight volumes.
These insights can guide strategic efforts in delay mitigation, particularly focusing on improving efficiency in short- and mid-range operations where the majority of delays are concentrated.
6.3. Summary of Findings
This classification analysis systematically explored five key flight delay causes—Carrier, Late Aircraft, NAS, Security, and Weather—across multiple operational dimensions. By using grouped aggregations and targeted visualizations, we identified the relative contributions and variation patterns of these causes under different scenarios:
- Carrier-related and Late Aircraft delays were consistently dominant, especially among regional airlines, high-traffic airports (e.g., DFW, ORD), and short-to-medium-haul flights.
- NAS delays exhibited strong time-of-day variation, peaking during midday and evening hours.
- Weather delays remained stable across most conditions but were slightly more visible in specific months and locations (e.g., ORD, LGA).
- Security delays were minimal in all scenarios, contributing little to total delay time.
Across dimensions such as month, airline, airport, hour of day, weekday, holiday status, and flight distance, our analysis provided a comprehensive view of how delay causes behave. These findings can support delay prediction, resource allocation, and operational planning in real-world aviation management systems.