setwd("C:/Users/juanc/OneDrive/Desktop/School/INFOH_510/Data_Dives") 
library(tidyverse)   
## Warning: package 'tidyverse' was built under R version 4.3.2
## Warning: package 'ggplot2' was built under R version 4.3.2
## Warning: package 'tibble' was built under R version 4.3.2
## Warning: package 'tidyr' was built under R version 4.3.2
## Warning: package 'readr' was built under R version 4.3.2
## Warning: package 'purrr' was built under R version 4.3.2
## Warning: package 'dplyr' was built under R version 4.3.2
## Warning: package 'stringr' was built under R version 4.3.2
## Warning: package 'forcats' was built under R version 4.3.2
## Warning: package 'lubridate' was built under R version 4.3.2
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.4
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.4.4     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.0
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(readr)
adc <- read_csv("Airline_Delay_Cause_2022.csv") #loading data 
## Rows: 20345 Columns: 20
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr  (4): carrier, carrier_name, airport, airport_name
## dbl (16): month, arr_flights, arr_del15, carrier_ct, weather_ct, nas_ct, sec...
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
adc_renamed <- read_csv("adc_2022_ed.csv") #loading data 
## Rows: 20345 Columns: 18
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr  (2): carrier, airport
## dbl (16): month, arrived_flights, total_delayed, air_carrier_delay, weather_...
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
adc_names <- read_csv("adc_names.csv") #loading data 
## Rows: 17 Columns: 2
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): carrier, carrier_name
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Week 5 Data Dive

In this week data dive our focus will be on the documentation of the data. Where the data comes from and how is encoded.

To view the documentation click on the link to the Bureau of Transportation.

As this data encompasses delays for flights, we must first understand what a delay is. According to the data documentation, “A flight is counted as”on time” if it operated less than 15 minutes later than the scheduled time shown in the carriers’ Computerized Reservations Systems (CRS). Arrival performance is based on arrival at the gate. Departure performance is based on departure from the gate. ” With this, we can confirm a delay is when a flight arrives or departs 15 minutes later than scheduled.

Before we begin, some renaming and documentation has taken place from week to week (largely in week 3). In this data dive all names will be renamed to their original nomenclature as of week 1.

In the original data, three columns will be chosen that are unclear. Research on why the original organization choose to encode the data that way through documentation.

After looking at the documentation, we will see if any column is still unclear. We can then visualize this columns variables to investigate what is the issue, what is unclear from the data, and why is it unclear.

We will finalize this weeks data dive be reviewing the risk from this columns has on the data and what we can do to reduce this risk for future analysis.

Unclear Columns

Renaming Columns

colnames(adc)
##  [1] "month"               "carrier"             "carrier_name"       
##  [4] "airport"             "airport_name"        "arr_flights"        
##  [7] "arr_del15"           "carrier_ct"          "weather_ct"         
## [10] "nas_ct"              "security_ct"         "late_aircraft_ct"   
## [13] "arr_cancelled"       "arr_diverted"        "arr_delay"          
## [16] "carrier_delay"       "weather_delay"       "nas_delay"          
## [19] "security_delay"      "late_aircraft_delay"

In the original data we see that carrier and airport have a short form and the full name. In week 3 & 4, the full name column were used to create a separate table (adc_name). This allowed us to group_by without the need to group by carrier, carrier_name, airport, and airport_name. When we analyzed the data, we can reference the adc_name table to view the full names.

The data only viewed by the column names can be confusing, as many of the columns are shortened.

head(adc)
## # A tibble: 6 × 20
##   month carrier carrier_name      airport airport_name     arr_flights arr_del15
##   <dbl> <chr>   <chr>             <chr>   <chr>                  <dbl>     <dbl>
## 1    12 9E      Endeavor Air Inc. ABE     Allentown/Bethl…          27         7
## 2    12 9E      Endeavor Air Inc. ABY     Albany, GA: Sou…          69         3
## 3    12 9E      Endeavor Air Inc. AEX     Alexandria, LA:…          62        14
## 4    12 9E      Endeavor Air Inc. AGS     Augusta, GA: Au…          47        10
## 5    12 9E      Endeavor Air Inc. ALB     Albany, NY: Alb…         131        34
## 6    12 9E      Endeavor Air Inc. ATL     Atlanta, GA: Ha…        1691       257
## # ℹ 13 more variables: carrier_ct <dbl>, weather_ct <dbl>, nas_ct <dbl>,
## #   security_ct <dbl>, late_aircraft_ct <dbl>, arr_cancelled <dbl>,
## #   arr_diverted <dbl>, arr_delay <dbl>, carrier_delay <dbl>,
## #   weather_delay <dbl>, nas_delay <dbl>, security_delay <dbl>,
## #   late_aircraft_delay <dbl>

Looking at the head of adc, we see that the delays are seen twice, once with the ending “_ct”, and the other with the ending “_delay”. Looking at the documentation on the website of the Bureau of Transportation, the “_ct” delays is count of delays, while “_delay” is the count of delays in minutes.

In the Week 3 Deep Dive I renamed these columns to clearly indicate what they contain.

colnames(adc_renamed) #adc_renmaed is a seperate table, created to rename the columns
##  [1] "month"                                 
##  [2] "carrier"                               
##  [3] "airport"                               
##  [4] "arrived_flights"                       
##  [5] "total_delayed"                         
##  [6] "air_carrier_delay"                     
##  [7] "weather_delay"                         
##  [8] "national_aviation_system_delay"        
##  [9] "security_delay"                        
## [10] "late_aircraft_delay"                   
## [11] "cancelled"                             
## [12] "diverted"                              
## [13] "total_delayed_minutes"                 
## [14] "air_carrier_delay_minutes"             
## [15] "weather_delay_minutes"                 
## [16] "national_aviation_system_delay_minutes"
## [17] "security_delay_minutes"                
## [18] "late_aircraft_delay _minutes"

Further Documentation Research

Now that we clearly have indicated the columns and what they represent, we must look into the columns and investigate each one.

We can understand what a flight is and what a delay is, however what exactly is a “air_carrier_delay”, “national_aviation_system_delay”, “late_aircraft_delay”. These encompass both delays counted and minutes delayed.

adc_renamed |> 
  select(air_carrier_delay, national_aviation_system_delay, late_aircraft_delay)
## # A tibble: 20,345 × 3
##    air_carrier_delay national_aviation_system_delay late_aircraft_delay
##                <dbl>                          <dbl>               <dbl>
##  1              4.37                           2.1                 0.53
##  2              0.61                           0.02                2.37
##  3              5.28                           3.42                0.96
##  4              5.12                           1                   2.31
##  5             12.1                            6.32               12.9 
##  6             61.5                           60.4               128.  
##  7              7.53                           7.83                2.26
##  8              0                              0.31                0.69
##  9              3.61                           1                   1.39
## 10              3.55                           6.45                0   
## # ℹ 20,335 more rows

Delay Documentation

Copied from the Bureau of Transportation:

  • Air Carrier: The cause of the cancellation or delay was due to circumstances within the airline’s control (e.g. maintenance or crew problems, aircraft cleaning, baggage loading, fueling, etc.).

  • National Aviation System (NAS): Delays and cancellations attributable to the national aviation system that refer to a broad set of conditions, such as non-extreme weather conditions, airport operations, heavy traffic volume, and air traffic control.

  • Late-arriving aircraft: A previous flight with same aircraft arrived late, causing the present flight to depart late.

These explanations are straight to the point and provide a complete picture of what these delays causes are.

However there is still some questions to be asked regarding the late aircraft delay cause.

Late Aircraft Delay

In this data, a flight can be delayed twice, once at take off and once at arrival. The data is compiled by Month, Airport, and Airline. Meaning within the n month, x airline, and y airport what were the delay numbers and causes of those delays.

However a late aircraft delay is a causality of another delay, either being late from the last leg, or an original delay such as weather or security. Late aircraft delay can be a domino effect affecting subsequent flights. This can potentially taint the data.

Visualization of Late Aircraft Delay

Using a code fro Week 4 Deep Dive, we can see how much more larger the late aircraft delay is compared to the others.

adc_renamed |> 
    group_by(month) |>
    summarize(
      total_flights = sum(arrived_flights, na.rm = TRUE), 
      total_delayed_flights = sum(total_delayed, na.rm = TRUE), 
      sum_late_ac = sum(late_aircraft_delay, na.rm = TRUE), 
      diff_late_to_total = total_delayed_flights - sum_late_ac, 
      ) |>
  ggplot(aes(x = month)) +
  geom_line(aes(y = total_delayed_flights, color = "Total Delayed Flights"), size = 1) +
  geom_line(aes(y = sum_late_ac, color = "Late Aircraft Delays"), size = 1) +
  geom_line(aes(y = diff_late_to_total, color = "Rest of Cause Delays"), size = 1) +
  labs(title = "Total Delayed Flights and Late Aircraft Delay by Month",
       y = "Count",
       x = "Month") +
  scale_x_continuous(breaks = 1:12, labels = month.abb)
## Warning: Using `size` aesthetic for lines was deprecated in ggplot2 3.4.0.
## ℹ Please use `linewidth` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

The late aircraft delays are a substantial part of the delays but may be caused by the other delays.

This data column is critical to view the whole picture of total delays in all airports and airlines in a given month. However, when attempting to identify the original causes and the frequency that they occur, this information is muddying the data.

In future analysis, a new column must be created removing the late aircraft delays when measuring the other causes in comparison the total delays.