final110khan

#Set working directory and read data-set

getwd()
[1] "/Users/thydakhan/Library/Mobile Documents/com~apple~CloudDocs/Data 101"
library(tidyverse)
Warning: package 'ggplot2' was built under R version 4.5.2
── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.1.4     ✔ readr     2.1.5
✔ forcats   1.0.1     ✔ stringr   1.5.2
✔ ggplot2   4.0.3     ✔ tibble    3.3.0
✔ lubridate 1.9.4     ✔ tidyr     1.3.1
✔ purrr     1.1.0     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
library(lubridate)
library(janitor)

Attaching package: 'janitor'

The following objects are masked from 'package:stats':

    chisq.test, fisher.test
library(plotly)
Warning: package 'plotly' was built under R version 4.5.2

Attaching package: 'plotly'

The following object is masked from 'package:ggplot2':

    last_plot

The following object is masked from 'package:stats':

    filter

The following object is masked from 'package:graphics':

    layout
flights<- read.csv("~/Library/Mobile Documents/com~apple~CloudDocs/Data 101/flight_edges.csv")
head(flights)
  MHK AMW Manhattan..KS Ames..IA X21 X30 X1 X254.0 X200810 X122049 X86219
1 EUG RDM    Eugene, OR Bend, OR  41 396 22    103  199011  284093  76034
2 EUG RDM    Eugene, OR Bend, OR  88 342 19    103  199012  284093  76034
3 EUG RDM    Eugene, OR Bend, OR  11  72  4    103  199010  284093  76034
4 MFR RDM   Medford, OR Bend, OR   0  18  1    156  199002  147300  76034
5 MFR RDM   Medford, OR Bend, OR  11  18  1    156  199003  147300  76034
6 MFR RDM   Medford, OR Bend, OR   2  72  4    156  199001  147300  76034
colnames(flights)
 [1] "MHK"           "AMW"           "Manhattan..KS" "Ames..IA"     
 [5] "X21"           "X30"           "X1"            "X254.0"       
 [9] "X200810"       "X122049"       "X86219"       

#Compute and fix header

flights <- readr::read_csv(
  "flight_edges.csv",
  col_names = FALSE
)
Rows: 3606803 Columns: 11
── Column specification ────────────────────────────────────────────────────────
Delimiter: ","
chr (11): X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11

ℹ Use `spec()` to retrieve the full column specification for this data.
ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
colnames(flights) <- c(
  "Origin",
  "Destination",
  "OriginCity",
  "DestinationCity",
  "Passengers",
  "Seats",
  "Flights",
  "Distance",
  "FlyDate",
  "OriginPopulation",
  "DestinationPopulation"
)
#clean function for names so coding is easier
flights <- flights %>%
  clean_names()
head(flights)
# A tibble: 6 × 11
  origin destination origin_city   destination_city passengers seats flights
  <chr>  <chr>       <chr>         <chr>            <chr>      <chr> <chr>  
1 MHK    AMW         Manhattan..KS Ames..IA         X21        X30   X1     
2 EUG    RDM         Eugene, OR    Bend, OR         41         396   22     
3 EUG    RDM         Eugene, OR    Bend, OR         88         342   19     
4 EUG    RDM         Eugene, OR    Bend, OR         11         72    4      
5 MFR    RDM         Medford, OR   Bend, OR         0          18    1      
6 MFR    RDM         Medford, OR   Bend, OR         11         18    1      
# ℹ 4 more variables: distance <chr>, fly_date <chr>, origin_population <chr>,
#   destination_population <chr>

#Introduction #Research question-

#How did the average load factor of U.S. domestic flights change between 1998 and 2009

#How did the distribution of median average yearly flight load differ from 1998-2003 and 2004-2009?

#Source- #####This analysis follows a dataset containing detailed information on all recorded U.S. domestic flights from 1998-2009 published by the US Census.

#######Citation: #Perkins, Jacob. 3.5 Million+ US Domestic Flights from 1990 to 2009. Infochimps, 2010, http://infochimps.org/datasets/d35-million-us-domestic-flights-from-1990-to-2009

#What is load factor? ###The load factor is a percentage based on the number of passengers per flight, dividied by the number of available seats.A higher load factor means more seats are filled, while a lower load factor means many seats are empty, suggesting underutilized flights. Analyzing the change in load factor is a great indication of the efficiency and profitability during economic hardship, and to better understand for future reference the determent a recession has on the travel habits within US airports.

#View raw data then convert it into character format to be able to be explored

glimpse(flights)
Rows: 3,606,803
Columns: 11
$ origin                 <chr> "MHK", "EUG", "EUG", "EUG", "MFR", "MFR", "MFR"…
$ destination            <chr> "AMW", "RDM", "RDM", "RDM", "RDM", "RDM", "RDM"…
$ origin_city            <chr> "Manhattan..KS", "Eugene, OR", "Eugene, OR", "E…
$ destination_city       <chr> "Ames..IA", "Bend, OR", "Bend, OR", "Bend, OR",…
$ passengers             <chr> "X21", "41", "88", "11", "0", "11", "2", "7", "…
$ seats                  <chr> "X30", "396", "342", "72", "18", "18", "72", "1…
$ flights                <chr> "X1", "22", "19", "4", "1", "1", "4", "1", "2",…
$ distance               <chr> "X254.0", "103", "103", "103", "156", "156", "1…
$ fly_date               <chr> "X200810", "199011", "199012", "199010", "19900…
$ origin_population      <chr> "X122049", "284093", "284093", "284093", "14730…
$ destination_population <chr> "X86219", "76034", "76034", "76034", "76034", "…
flights <- flights %>%
  mutate(
    fly_date = as.character(fly_date),
    fly_date = paste0(fly_date, "01"),
    fly_date = ymd(fly_date)
  )
Warning: There was 1 warning in `mutate()`.
ℹ In argument: `fly_date = ymd(fly_date)`.
Caused by warning:
!  1 failed to parse.

#Fix warning and locate the one value could not be properly converted

flights %>%
  filter(is.na(fly_date)) %>%
  select(fly_date)
# A tibble: 1 × 1
  fly_date
  <date>  
1 NA      

#Filter to keep only applicable data

flights %>%
  filter(is.na(fly_date))
# A tibble: 1 × 11
  origin destination origin_city   destination_city passengers seats flights
  <chr>  <chr>       <chr>         <chr>            <chr>      <chr> <chr>  
1 MHK    AMW         Manhattan..KS Ames..IA         X21        X30   X1     
# ℹ 4 more variables: distance <chr>, fly_date <date>, origin_population <chr>,
#   destination_population <chr>

#Check data summary

summary(flights$fly_date)
        Min.      1st Qu.       Median         Mean      3rd Qu.         Max. 
"1990-01-01" "1996-03-01" "2001-11-01" "2001-01-11" "2006-01-01" "2009-12-01" 
        NA's 
         "1" 

#Clean both passenger and seat data

flights_clean <- flights %>%
  filter(
    !is.na(fly_date)
  )

flights_clean <- flights_clean %>%
  filter(
    !is.na(passengers),
    passengers > 0,
    !is.na(seats),
    seats > 0
  )

I had to fix the error of the console reading my data as numeric when all values should be read as characters, I went from about 3.6 million original rows to 3.2 million rows after cleaning

flights_clean <- flights_clean %>%
  mutate(
    passengers = as.numeric(passengers),
    seats = as.numeric(seats),
    flights = as.numeric(flights),
    distance = as.numeric(distance),
    origin_population = as.numeric(origin_population),
    destination_population = as.numeric(destination_population)
  )

glimpse(flights_clean)
Rows: 3,220,900
Columns: 11
$ origin                 <chr> "EUG", "EUG", "EUG", "MFR", "MFR", "MFR", "MFR"…
$ destination            <chr> "RDM", "RDM", "RDM", "RDM", "RDM", "RDM", "RDM"…
$ origin_city            <chr> "Eugene, OR", "Eugene, OR", "Eugene, OR", "Medf…
$ destination_city       <chr> "Bend, OR", "Bend, OR", "Bend, OR", "Bend, OR",…
$ passengers             <dbl> 41, 88, 11, 11, 2, 7, 7, 8, 453, 784, 749, 11, …
$ seats                  <dbl> 396, 342, 72, 18, 72, 18, 36, 18, 3128, 2720, 2…
$ flights                <dbl> 22, 19, 4, 1, 4, 1, 2, 1, 23, 20, 22, 1, 23, 16…
$ distance               <dbl> 103, 103, 103, 156, 156, 156, 156, 228, 228, 22…
$ fly_date               <date> 1990-11-01, 1990-12-01, 1990-10-01, 1990-03-01…
$ origin_population      <dbl> 284093, 284093, 284093, 147300, 147300, 147300,…
$ destination_population <dbl> 76034, 76034, 76034, 76034, 76034, 76034, 76034…

#Create load factor

flights_clean <- flights_clean %>%
  mutate(
    load_factor = passengers / seats
  )
summary(flights_clean$load_factor)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
4.789e-04 5.101e-01 6.647e-01 6.432e-01 8.006e-01 1.106e+01 

#Create data table for year

flights_clean <- flights_clean %>%
  mutate(
    year = year(fly_date)
  )
summary(flights_clean$year)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1990    1995    2001    2000    2005    2009 

#We see there are 270 more passangers than seats and should further explore

sum(flights_clean$load_factor > 1, na.rm = TRUE)
[1] 270
sum(flights_clean$load_factor < 0, na.rm = TRUE)
[1] 0

#Explore how to fix this issue, in order to do so I must explore the culprit of this issue, whether it be thats the seats are recorded across entire cabin capacity or if passanger data is inconsistently recorded.

flights_clean %>%
  filter(load_factor > 1) %>%
  select(
    origin,
    destination,
    passengers,
    seats,
    flights,
    distance,
    fly_date,
    load_factor
  ) %>%
  head(10)
# A tibble: 10 × 8
   origin destination passengers seats flights distance fly_date   load_factor
   <chr>  <chr>            <dbl> <dbl>   <dbl>    <dbl> <date>           <dbl>
 1 LAS    RNO                179   163       1      345 1990-01-01        1.10
 2 TPA    MIA                250   179       1      204 1992-03-01        1.40
 3 JFK    MIA                681   680       4     1090 1993-07-01        1.00
 4 JFK    MIA                167   137       1     1090 1994-06-01        1.22
 5 JFK    MIA                158   137       1     1090 1994-07-01        1.15
 6 JFK    MIA                175   168       1     1090 1996-07-01        1.04
 7 TPA    MIA                366   360       1      204 1996-10-01        1.02
 8 FLL    TPA               1386  1080       4      197 1990-07-01        1.28
 9 FLL    TPA               1228  1080       4      197 1990-04-01        1.14
10 FLL    TPA               1247  1080       4      197 1990-10-01        1.15

#In running the summary stats of the data set with this 270 concerning obersevations , one can see how extreme of a potential skew these data points have on the overall distriubtion.

summary(
  flights_clean$load_factor[
    flights_clean$load_factor > 1
  ]
)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.001   1.036   1.085   1.146   1.120  11.057 

#Choose to omit based on how extreme it skewed the datas load factors all over 100% and noteably inflating the maximum value. Then check to ensure the load factor has been properly cleaned and values omitted.

flights_clean <- flights_clean %>%
  filter(
    load_factor <= 1
  )
sum(flights_clean$load_factor > 1, na.rm = TRUE)
[1] 0

#Restrict my analysis to the desired time period of 1998-2009

flights_clean <- flights_clean %>%
  filter(
    year >= 1998,
    year <= 2009
  )

flights_clean <- flights_clean %>%
  mutate(
    period = if_else(
      year < 2008,
      "Before 2008",
      "2008-2009"
    )
  )

summary(flights_clean$year)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1998    2001    2004    2004    2007    2009 

#Calculate the yearly load factor from the desired years

yearly_load <- flights_clean %>%
  group_by(year) %>%
  summarise(
    total_passengers = sum(passengers, na.rm = TRUE),
    total_seats = sum(seats, na.rm = TRUE)
  ) %>%
  mutate(
    load_factor = total_passengers / total_seats
  ) %>%
  arrange(year)

yearly_load
# A tibble: 12 × 4
    year total_passengers total_seats load_factor
   <dbl>            <dbl>       <dbl>       <dbl>
 1  1998        491915904   740059536       0.665
 2  1999        511336310   772257020       0.662
 3  2000        533957069   792445849       0.674
 4  2001        497009691   759553134       0.654
 5  2002        486021543   731002404       0.665
 6  2003        508988911   741146656       0.687
 7  2004        546469268   776053839       0.704
 8  2005        568207257   782392392       0.726
 9  2006        564727437   751970874       0.751
10  2007        579257475   762676792       0.760
11  2008        554997072   737437088       0.753
12  2009        526301558   689591729       0.763
#Calculating Total passengers ÷ total seats for each year to find  the overall seat utilization for each year

#Turn calculation into percentage

yearly_load <- yearly_load %>%
  mutate(
    load_factor_percent = load_factor * 100
  )

yearly_load
# A tibble: 12 × 5
    year total_passengers total_seats load_factor load_factor_percent
   <dbl>            <dbl>       <dbl>       <dbl>               <dbl>
 1  1998        491915904   740059536       0.665                66.5
 2  1999        511336310   772257020       0.662                66.2
 3  2000        533957069   792445849       0.674                67.4
 4  2001        497009691   759553134       0.654                65.4
 5  2002        486021543   731002404       0.665                66.5
 6  2003        508988911   741146656       0.687                68.7
 7  2004        546469268   776053839       0.704                70.4
 8  2005        568207257   782392392       0.726                72.6
 9  2006        564727437   751970874       0.751                75.1
10  2007        579257475   762676792       0.760                76.0
11  2008        554997072   737437088       0.753                75.3
12  2009        526301558   689591729       0.763                76.3

#Create data table to show the percent change year to year

yearly_load <- yearly_load %>%
  mutate(
    rate_of_change = (load_factor - lag(load_factor)) / lag(load_factor) * 100
  )

yearly_load <- yearly_load %>%
  mutate(
    percentage_point_change = 
      (load_factor - lag(load_factor)) * 100
  )

#Visualization 1 — Yearly Load Factor + Year-to-Year Rate of Change

yearly_plot <- ggplot(
  yearly_load, 
  aes(
    x = year, 
    y = load_factor_percent,
    text = paste(
      "Year:", year,
      "<br>Load Factor:", round(load_factor_percent, 2), "%",
      "<br>Year-to-Year Change:", round(rate_of_change, 2), "%"
    )
  )
) +
  geom_line(
    aes(color = "Load Factor"), 
    linewidth = 1.2
  ) +
  geom_point(
    aes(color = "Load Factor"), 
    size = 3
  ) +
  geom_point(
    data = yearly_load %>% filter(year >= 2008), 
    aes(color = "2008–2009"), 
    size = 2
  ) +
  geom_vline(
    xintercept = 2008, 
    linetype = "dashed", 
    aes(color = "2008 Financial Crisis"), 
    linewidth = 1
  ) +
  annotate(
    "text", 
    x = 2008.5, 
    y = max(yearly_load$load_factor_percent) - 1, 
    label = "2008 Financial Crisis", 
    hjust = 0, 
    size = 4
  ) +
  scale_color_manual(
    values = c(
      "Load Factor" = "#00BFFF", 
      "2008–2009" = "deeppink4",           
      "2008 Financial Crisis" = "#C1FFC1" 
    )
  ) +
  labs(
    title = "U.S. Domestic Flight Load Factor from 1998–2009",
    subtitle = "Yearly seat utilization and trend leading to and right after 2008 financial crisis",
    x = "Year",
    y = "Load Factor (%)",
    color = "Category",
    caption = "Source: Perkins, Jacob. 3.5 Million+ US Domestic Flights from 1990 to 2009, Infochimps (2010)."
  ) +
  theme_dark() +
  theme(
    plot.title = element_text(face = "italic", size = 12),
    plot.subtitle = element_text(size = 11),
    legend.position = "bottom"
  )
Warning: `geom_vline()`: Ignoring `mapping` because `xintercept` was provided.
yearly_plot
`geom_line()`: Each group consists of only one observation.
ℹ Do you need to adjust the group aesthetic?

#Visualization 2 — Distribution of median load factors from 1998-2003 and 2004-2009

Refine load factor data to a specific time period and review summary stats

summary(flights_clean$load_factor)
     Min.   1st Qu.    Median      Mean   3rd Qu.      Max. 
0.0005163 0.5628415 0.7083150 0.6811566 0.8298518 1.0000000 
flights_clean %>%
  group_by(period) %>% summarise(
    minimum = min(load_factor, na.rm = TRUE),
    q1 = quantile(load_factor, 0.25, na.rm = TRUE),
    median = median(load_factor, na.rm = TRUE),
    q3 = quantile(load_factor, 0.75, na.rm = TRUE),
    maximum = max(load_factor, na.rm = TRUE),
    mean = mean(load_factor, na.rm = TRUE) )
# A tibble: 2 × 7
  period       minimum    q1 median    q3 maximum  mean
  <chr>          <dbl> <dbl>  <dbl> <dbl>   <dbl> <dbl>
1 2008-2009   0.000925 0.628  0.765 0.868       1 0.728
2 Before 2008 0.000516 0.551  0.695 0.819       1 0.671

#Create yearly median data set

yearly_medians <- flights_clean %>%
  group_by(year) %>%
  summarise( median_load_factor = median(load_factor, na.rm = TRUE)
 ) %>%
  mutate(  median_percent = median_load_factor * 100,
    period = if_else( year <= 2003,
 "1998–2003",
      "2004–2009"
    )
  )
yearly_medians
# A tibble: 12 × 4
    year median_load_factor median_percent period   
   <dbl>              <dbl>          <dbl> <chr>    
 1  1998              0.661           66.1 1998–2003
 2  1999              0.653           65.3 1998–2003
 3  2000              0.666           66.6 1998–2003
 4  2001              0.647           64.7 1998–2003
 5  2002              0.659           65.9 1998–2003
 6  2003              0.675           67.5 1998–2003
 7  2004              0.698           69.8 2004–2009
 8  2005              0.723           72.3 2004–2009
 9  2006              0.756           75.6 2004–2009
10  2007              0.766           76.6 2004–2009
11  2008              0.759           75.9 2004–2009
12  2009              0.772           77.2 2004–2009

#Visualization 2 Distribution of sub set medians compared

ggplot(yearly_medians, aes(
  x = period,
  y = median_percent,
  fill = period
)) +
  geom_boxplot(
    width = 0.7,
    alpha = 0.82,
    outlier.shape = NA
  ) +
  geom_jitter(
    aes(color = period),
    width = 0.11,
    size = 2.6
  ) +
  scale_fill_manual(
    values = c(
      "1998–2003" = "#2C7FB8",
      "2004–2009" = "#556B2F"
    )
  ) +
  scale_color_manual(
    values = c(
      "1998–2003" = "#A2CD5A",
      "2004–2009" = "#238443"
    )
  ) +
  labs(
    title = "Distribution of Yearly Median U.S. Flight Load Factors",
    subtitle = "Comparing 1998–2003 to 2004–2009",
    x = "Time Period",
    y = "Median Load Factor (%)",
    fill = "Time Period",
    color = "Yearly Median",
    caption = "Source: Perkins, Jacob. 3.5 Million+ US Domestic Flights from 1990 to 2009, Infochimps (2010)."
  ) +
  annotate(
    "text",
    x = 2,
    y = max(yearly_medians$median_percent, na.rm = TRUE) + 1,
    label = "Later period",
    fontface = "italic"
  ) +
  theme_minimal() +
  theme(
    plot.title = element_text(face = "italic", size = 11),
    plot.subtitle = element_text(size = 9),
    legend.position = "bottom"
  )

#Calculate the average difference in medians for both time periods

period_summary <- yearly_medians %>% group_by(period) %>% summarise( average_median = mean(median_percent), .groups = "drop" )
period_summary
# A tibble: 2 × 2
  period    average_median
  <chr>              <dbl>
1 1998–2003           66.0
2 2004–2009           74.6

#Findings- #You can see the cluster from 1998-2003 is much more rigid than that of 2004-200

#74.58 − 66.04 = 8.54 percentage points

###The average yearly median increased by 8.54 percentage points between 1998-2003 and 2004-2009.

#Conclusion-

#The results were somewhat surprising because the data does not show an obvious decline in load factor during the later part of the study period like I hypothesized it would have due to such a stark drop in the economy. In fact, the median load factor was generally higher in 2004–2009 than in 1998–2003. This means that data as such cannot simply be used as the sole visualization to see the effects of immediate changes within the economy in industries such as aviation, rather a tool to watch how over the long haul, depressions may affect distributuions over longer time periods.

#One limitation one must keep in mind is the flight industry capacity has tripled since the 1990’s when accounting for that time periods datas dsitrubtions. As well that this analysis only examines load factor and does not include other varying factors that could influence airline travel, such as ticket prices,, fuel costs, route distance, airline type, or the number of flights in the air. Because of this, the results show an association and change over time , not a correlation. With more time, I would include economic and airline-specific variables to investigate how changes in the economy may have influenced passenger travel and flight occupancy. Future research can build on this exploration to further see how a roust market like the UNitef States may tend to trend toward in economic recessions.

#By continuing to monitor changes over time, researchers can identify how events such as economic recessions, pandemics directly affect flight occupancy rates, and overall revenue loss that would occur in the modern aviation system.