In this section you will examine the impact of some variables departure delay. Please consider that some values of dep_delay are NA so you need to consider it when computing the mean of departure delay. Choose the adequate plot for each variable.
1.1 Planned Departure Hour
For each origin airport, examine the evolution of average departure delay for each hour of planned departure, represented with the variable hour. Comment briefly the obtained result.
We can conclude that the number of delays are higher in hours 16 - 19.
There is a trend which tell us that during the beginning of the day and the end of day the delays are lower.
We can also observe that Airport LGA and JFK perform better than Airport EWR in terms of the average of delays per hour.
1.2 Number of Flights
For each origin and for each value of time_hour, calculate the average departure delay and the number of flights. With those values, plot for each origin airport the average departure delay as a function of number of flights. Comment briefly the obtained result.
plot1.2_EWR <-ggplot(by_EWR_total, aes(Total_Flights, AverageDep_Delay)) +geom_point() +geom_smooth(method ="lm") +##gam se adapta melhor aos dadoslabs(x ="Total Number of flights", y ="Average Departure Delays",title ="Total Number of flights vs Average Departure Delays (EWR)") +theme_minimal()+theme(legend.position ="none",plot.title =element_text(hjust ="0.5"))plot1.2_LGA <-ggplot(by_LGA_total, aes(Total_Flights, AverageDep_Delay)) +geom_point() +geom_smooth(method ="lm") +labs(x ="Total Number of flights", y ="Average Departure Delays",title ="Total Number of flights vs Average Departure Delays (LGA)") +theme_minimal()+theme(legend.position ="none",plot.title =element_text(hjust ="0.5"))plot1.2_JFK <-ggplot(by_JFK_total, aes(Total_Flights, AverageDep_Delay)) +geom_point() +geom_smooth(method ="lm") +labs(x ="Total Number of flights", y ="Average Departure Delays",title ="Total Number of flights vs Average Departure Delays (JFK)") +theme_minimal()+theme(legend.position ="none",plot.title =element_text(hjust ="0.5"))plot1.2_EWR
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
`geom_smooth()` using formula = 'y ~ x'
Based on the above graphics, it is clear that as the number of flights increases, the average departure delay time also increases.
1.3 Weekday
Create a weekday column from time_hour using the wday() function of lubridate with label = TRUE. For each day of the and for each origin, calculate the average departure delay. Comment briefly the obtained result.
flights_weekday <- flights %>%mutate(weekday =wday(time_hour, label =TRUE))average_delays <- flights_weekday %>%filter(!is.na(dep_delay)) %>%group_by(origin, weekday) %>%summarise(average_delay =mean(dep_delay, na.rm =TRUE), .groups ='drop')ggplot(average_delays, aes(x = weekday, y = average_delay, fill = origin)) +geom_col(position ="dodge") +labs(x ="Weekday", y ="Average Departure Delay (minutes)",title ="Average Departure Delays by Weekday and Origin") +theme_minimal()
All three airports experience the highest average departure delays on Thursdays and Fridays.
EWR consistently shows higher departure delays than JFK and LGA (higher total number of flights)
Departure delays are generally lower on weekends (Sunday and Saturday) across all airports.
1.4 Month
For each origin airport, examine with an adequate plot the evolution of the average departure delay for each month. Comment briefly the obtained result.
JFK_delays_mean <-mean(JFK_delay_data$TotalDelays)plot1.3_EWR <-ggplot(EWR_delay_data, aes(factor(Month), TotalDelays, color =factor(Month), group =2)) +geom_point() +geom_line(color ="red") +geom_hline(yintercept = EWR_delays_mean, color="black") +#scale_fill_manual(values = c("red", "blue", "green")) we need 12labs(x="Month", y="Delays",title ="Month vs Delays (EWR)") +theme_minimal(base_size =11) +theme(legend.position ="none") plot1.3_LGA <-ggplot(LGA_delay_data, aes(factor(Month), TotalDelays, color =factor(Month), group =2)) +geom_point() +geom_line(color ="red") +geom_hline(yintercept = LGA_delays_mean, color="black") +#scale_fill_manual(values = c("red", "blue", "green")) we need 12labs(x="Month", y="Delays",title ="Month vs Delays (LGA)") +theme_minimal(base_size =11) +theme(legend.position ="none") plot1.3_JFK <-ggplot(JFK_delay_data, aes(factor(Month), TotalDelays, color =factor(Month), group =2)) +geom_point() +geom_line(color ="red") +geom_hline(yintercept = JFK_delays_mean, color="black") +#scale_fill_manual(values = c("red", "blue", "green")) we need 12labs(x="Month", y="Delays",title ="Month vs Delays (JFK)") +theme_minimal(base_size =11) +theme(legend.position ="none") grid.arrange(plot1.3_EWR, plot1.3_LGA, plot1.3_JFK, ncol=3)
We can conclude that the number of delays are higher in december and lowest in Setember, Octuber and November.
There is a trend which tell us that during the holiday season the delays are higher and they are lower just before the holiday period.
We can also observe that Airport LGA and JFK perform better than Airport EWR in terms of the average of delays per month. This observation can be reasoned out as EWR flies out more flights than LGA or JFK.
1.5 Wind
Join the flights and weather datasets, so that we can obtain the weather for each flight.
### Bringing the flights data to similar granularity of the weather databy_time_hour_airport = flights %>%filter(dep_delay >0) %>%group_by(origin, time_hour) %>%summarise(totalDelay =mean(dep_delay),totalflightsdelayed =n())
`summarise()` has grouped output by 'origin'. You can override using the
`.groups` argument.
merged_fli_weather <-inner_join(by_time_hour_airport, weather_data, by =c("origin","time_hour")) merged_fli_weather
Create a windy variable equal to one if wind_speed is larger than 25 and zero otherwise. For each origin airport, use boxplots to check if wind speed has impact on the average departure delay for each value of time_hour.
Let’s define a predictive model of delays for the EWR airport to predict if a flight will arrive late, that is, with a delay equal or larger than 15 minutes.
2.1 Target Variable
Filter the flights departing from EWR, excluding observations with NA in dep_delay. Define the target variable as a binary factor, being the positive case that the flight arrives late.
I chose the decision tree model due to its higher sensitivity, which ensures better identification of delayed flights, aligning with the primary objective of minimizing missed detections of departure delays.
2.7 Select and Fit Model
Select which model is best and train it with the whole train test. Briefly comment why have you chosen the model.
best_model <- dt_wf %>%fit(train_data)
2.8 Test the Final Model
Test the final model on the test set, obtaining the confusion matrix and the performance metrics. Comment briefly the results.
Sensitivity is expected to be high, indicating that the model is effective in identifying delayed flights. Conclusion - These results confirm the model’s alignment with the primary objective of detecting delayed flights effectively.