Flights Data Exploration¶
A real-world dataset containing flights data from the US Department of Transportation will be explored.
Loading and viewing the data.
In [1]:
import pandas as pd
import scipy.stats as stats
# Plots will be displayed inline
%matplotlib inline
from matplotlib import pyplot as plt
In [2]:
# Read dataset
flights = pd.read_csv('data/flights.csv')
flights.head()
# Dimensions
flights.shape
Out[2]:
The dataset contains observations of US domestic flights in 2013, and consists of the following fields:
- Year: The year of the flight (all records are from 2013)
- Month: The month of the flight
- DayofMonth: The day of the month on which the flight departed
- DayOfWeek: The day of the week on which the flight departed - from 1 (Monday) to 7 (Sunday)
- Carrier: The two-letter abbreviation for the airline.
- OriginAirportID: A unique numeric identifier for the departure aiport
- OriginAirportName: The full name of the departure airport
- OriginCity: The departure airport city
- OriginState: The departure airport state
- DestAirportID: A unique numeric identifier for the destination aiport
- DestAirportName: The full name of the destination airport
- DestCity: The destination airport city
- DestState: The destination airport state
- CRSDepTime: The scheduled departure time
- DepDelay: The number of minutes departure was delayed (flight that left ahead of schedule have a negative value)
- DelDelay15: A binary indicator that departure was delayed by more than 15 minutes (and therefore considered "late")
- CRSArrTime: The scheduled arrival time
- ArrDelay: The number of minutes arrival was delayed (flight that arrived ahead of schedule have a negative value)
- ArrDelay15: A binary indicator that arrival was delayed by more than 15 minutes (and therefore considered "late")
- Cancelled: A binary indicator that the flight was cancelled
Exploration of the flight data to analyze possible factors that affect delays in departure or arrival of a flight.
- Start by cleaning the data.
- Identify any null or missing data, and impute appropriate replacement values.
- Identify and eliminate any outliers in the DepDelay and ArrDelay columns.
- Explore the cleaned data.
- View summary statistics for the numeric fields in the dataset.
- Determine the distribution of the DepDelay and ArrDelay columns.
- Use statistics, aggregate functions, and visualizations to answer the following questions:
- What are the average (mean) departure and arrival delays?
- How do the carriers compare in terms of arrival delay performance?
- Is there a noticable difference in arrival delays for different days of the week?
- Which departure airport has the highest average departure delay?
- Do late departures tend to result in longer arrival delays than on-time departures?
- Which route (from origin airport to destination airport) has the most late arrivals?
- Which route has the highest average arrival delay?
In [3]:
# Checking null and missing data
flights.isnull().sum()
Out[3]:
There are 2761 null values in the variable DepDel15.
In [4]:
# Checking some lines with null values
flights[flights.isnull().any(axis=1)][['DepDelay','DepDel15']]
Out[4]:
In [5]:
flights[flights.isnull().any(axis=1)].DepDelay.describe()
Out[5]:
NaN values in DepDel15 are from DepDelay with value 0. It is considered that if DepDelay is 0, then the flight was not delayed, so DepDelay15 will be filled with 0.
In [6]:
# Filling NA values in DepDel15 with 0
flights.DepDel15 = flights.DepDel15.fillna(0)
# Checking null and missing data again
flights.isnull().sum()
flights.head()
Out[6]:
View summary statistics for the numeric fields in the dataset.¶
In [7]:
flights.describe()
Out[7]:
Determine the distribution of the DepDelay and ArrDelay columns.¶
In [11]:
# Create a function showing distribution
def show_distribution(var_data):
from matplotlib import pyplot as plt
# Get statistics
min_val = var_data.min()
max_val = var_data.max()
mean_val = var_data.mean()
med_val = var_data.median()
mod_val = var_data.mode()[0]
print('Minimum:{:.2f}\nMean:{:.2f}\nMedian:{:.2f}\nMode:{:.2f}\nMaximum:{:.2f}\n'.format(min_val,
mean_val,
med_val,
mod_val,
max_val))
# Create a figure for 2 subplots (2 rows, 1 column)
fig, ax = plt.subplots(2, 1, figsize = (10,4))
# Plot the histogram
ax[0].hist(var_data)
ax[0].set_ylabel('Frequency')
# Add lines for the mean, median, and mode
ax[0].axvline(x=min_val, color = 'gray', linestyle='dashed', linewidth = 2)
ax[0].axvline(x=mean_val, color = 'cyan', linestyle='dashed', linewidth = 2)
ax[0].axvline(x=med_val, color = 'red', linestyle='dashed', linewidth = 2)
ax[0].axvline(x=mod_val, color = 'yellow', linestyle='dashed', linewidth = 2)
ax[0].axvline(x=max_val, color = 'gray', linestyle='dashed', linewidth = 2)
# Plot the boxplot
ax[1].boxplot(var_data, vert=False)
ax[1].set_xlabel('Value')
# Add a title to the Figure
fig.suptitle('Data Distribution')
# Show the figure
fig.show()
In [12]:
# Get the variable to examine
col = flights['DepDelay']
# Call the function
show_distribution(col)
In [13]:
# Get the variable to examine
col = flights['ArrDelay']
# Call the function
show_distribution(col)
Identify and eliminate any outliers in the DepDelay and ArrDelay columns¶
In [14]:
# Removing Outliers - between 0.01 and 90 percentile
q90 = flights.DepDelay.quantile(0.90)
q01 = flights.DepDelay.quantile(0.01)
# Removing Outliers - between 0.01 and 90 percentile
q90a = flights.ArrDelay.quantile(0.90)
q01a = flights.ArrDelay.quantile(0.01)
# Filtered flights
flights = flights[(flights.DepDelay>q01) & (flights.DepDelay<q90)]
flights = flights[(flights.ArrDelay>q01a) & (flights.ArrDelay<q90a)]
flights.describe()
delayFields = ['DepDelay','ArrDelay']
# View the revised distributions
for col in delayFields:
show_distribution(flights[col])
In [15]:
# Number of Flights by Day of Week
print(flights.groupby(flights.DayOfWeek).Year.count())
There are less flights on Saturdays. On Sat the number of flights is smaller.
In [16]:
# Cancelled flights
pass_counts = flights['Cancelled'].value_counts()
plt.pie(pass_counts, labels=pass_counts)
plt.legend(pass_counts.keys().tolist())
Out[16]:
In [17]:
# Mean Dep Delay and Arr Delay by Day of Week
print(flights.groupby(flights.DayOfWeek)[delayFields].mean())
for col in delayFields:
flights.boxplot(column=col, by='DayOfWeek', figsize=(8,8))
How do the carriers compare in terms of arrival delay performance?¶
In [18]:
print(flights.groupby(flights.Carrier)[delayFields].mean())
for col in delayFields:
flights.boxplot(column=col, by='Carrier', figsize=(8,8))
Which departure airport has the highest average departure delay?¶
In [19]:
departure_airport_group = flights.groupby(flights.OriginAirportName)
mean_departure_delays = pd.DataFrame(departure_airport_group['DepDelay'].mean()).sort_values('DepDelay', ascending=False)
mean_departure_delays.plot(kind = "bar", figsize=(12,12))
mean_departure_delays
Out[19]:
Do late departures tend to result in longer arrival delays than on-time departures?¶
In [20]:
flights.boxplot(column='ArrDelay', by='DepDel15', figsize=(12,12))
Out[20]:
Which route (from origin airport to destination airport) has the most late arrivals?¶
In [24]:
# Add a routes column
routes = pd.Series(flights['OriginAirportName'] + ' -> ' + flights['DestAirportName'])
flights = pd.concat([flights, routes.rename("route")], axis=1)
# Sum arr delays per route
routearrdel = flights.groupby(flights.route)
pd.DataFrame(routearrdel['ArrDel15'].sum()).sort_values('ArrDel15', ascending=False)
Out[24]:
Which route has the highest average arrival delay?¶
In [25]:
# get the mean value and order from higher to minor
pd.DataFrame(routearrdel['ArrDelay'].mean()).sort_values('ArrDelay', ascending=False)
Out[25]:
In [23]:
# Checking DepDelay of cancelled flights
flights.boxplot(column='DepDelay', by='Cancelled', figsize=(8,5))
Out[23]:
