for (i in seq_along(airline_list)) {
url<-paste(base_url,airline_list[i],sep = "")
all_airline_urls<-c(all_airline_urls,url)}Flight Sentiment: Analyzing Passenger Reviews of Top North American Airlines
Introduction
In the contemporary business landscape, where consumer sentiment holds considerable influence, understanding the essence of customer reviews has become indispensable across various sectors. Airlines, naturally, are no exception to this rule. Deciphering and interpreting passenger feedback can offer invaluable insights into areas of strength and improvement within the aviation industry.
This project entails a comprehensive analysis of reviews for the top 10 airlines in North America, sourced directly from the reputable platform, airlinequality.com. Sophisticated data-driven techniques such as AFINN, BING, and NRC lexicons are employed to meticulously scrutinize each comment, thereby deriving a nuanced sentiment score.
Top 10 North American Airlines
The selection of the 10 airlines for this analysis was based on the annual number of passengers they carried. The chosen airlines and their country are as follows:
Air Canada (Canada)
Allegiant Air (United States)
American Airlines (United States)
Delta Airlines (United States)
Frontier Airlines (United States)
Jetblue Airways (United States)
Southwest Airlines (United States)
Spirit Airlines (United States)
United Airlines (United States)
Volaris (Mexico)
Data Scrape
Before commencing the analysis, the initial step involved scraping data from the webpages containing the airline reviews. Leveraging the structured URL format of airlinequality.com, which conveniently included the name of each airline in the URL, facilitated the automation of data retrieval.
URL Loop
Thankfully, the URL structure of airlinequality.com facilitated easy access to the review landing page for each airline. Each review page commenced with https://www.airlinequality.com/airline-reviews/ followed by the airline’s name separated by a dash. For instance, the URL for American Airlines would appear as: https://www.airlinequality.com/airline-reviews/american-airlines. To avoid the manual formatting of each URL, the following loop was employed:
This loop generated a list of valid airlinequality.com URLs for each airline in the dataset.
Review Loop
After storing the URLs in a list, another loop can be implemented to scrape each individual review. Each page yielded ten reviews per airline. These reviews were stored in a single column and varied in length. The subsequent loop, as shown below, facilitated this scraping process:
all_comments <- data.frame()
for (i in seq_along(all_airline_urls)) {
airline_url<-all_airline_urls[i]
print(paste("Collecting page",i,"of",length(all_airline_urls),":) ",sep =" "))
Sys.sleep(runif(1, 10, 15))
comments <- read_html(airline_url) %>%
html_elements("div.tc_mobile") %>%
html_elements("div.text_content") %>%
html_text2() %>%
str_remove(".*\\|\\s*") %>%
tibble(comment=.,airline=airline_list[which(all_airline_urls==airline_url)])
all_comments <- bind_rows(all_comments, comments)
print(paste(airline_url,"collected",sep = " "))
print(paste(nrow(all_comments),"total reviews collected so far!",sep=" "))}Initially, an empty data frame was instantiated to accommodate the reviews. Subsequently, within each iteration of the loop, the html_elements function captured the entirety of the review. Given the presence of unnecessary emojis on the page, the str_remove function was employed to eliminate all characters preceding the start of each review. Additionally, another column was added to track which airline was matched up to the review. Finally, the reviews were appended to the all_comments data frame, initiating the process anew for subsequent iterations.
Data Wrangling
In order to begin the sentiment analysis process, splitting the reviews into its own unique data frame was necessary. The following loop was used to do so:
for (i in seq_along(airline_list)){
airline_name_clean<-tolower(gsub(" ","_",airline_list[i]))
df_name<-paste(airline_name_clean,"_reviews",sep="")
current_airline_reviews<-airline_reviews %>%
filter(airline==airline_list[i]) %>%
select(comment)
assign(df_name,current_airline_reviews)}Word Splitting Loop
To see each review at a word level, the following loop was used to create a new data frame for each airline with each word at its own row:
for (df_name in ls(pattern = "_reviews$")) {
# Get df name
df <- get(df_name)
# Create new data frame for each airline at word level
df <- df %>%
mutate(comment_number = row_number()) %>%
unnest_tokens(word, comment) %>%
anti_join(stop_words)
assign(df_name, df)}Afinn Value
To get the Afinn score for each review for each airline, the following loop was used
for (df_name in ls(pattern = "_reviews$")) {
# Extract the data frame
df <- get(df_name)
# Get the airline name from the data frame name and format it
airline_name <- gsub("_reviews", "", df_name)
airline_name <- gsub("_", " ", airline_name)
airline_name <- str_to_title(airline_name)
# Perform inner join with afinn
df <- inner_join(df, afinn) %>%
group_by(comment_number) %>%
summarise(final_afinn_value = sum(value)) %>%
# Mutate the airline name
mutate(airline = airline_name)
# Assign the modified data frame back to its original name
assign(df_name, df)}In the end, the resulting data frame looked like this, for each airline:
| comment_number | final_afinn_value | airline |
|---|---|---|
| 1 | 13 | Air Canada |
| 2 | -14 | Air Canada |
| 3 | -5 | Air Canada |
| 4 | -2 | Air Canada |
| 5 | -4 | Air Canada |
| 6 | -1 | Air Canada |
| 7 | -2 | Air Canada |
| 8 | -1 | Air Canada |
| 9 | -3 | Air Canada |
| 10 | -9 | Air Canada |
On the Afinn scale, the higher positive the score, the more positive the review is.
Bing and NRC Sentiment Values
Similar processes took place in order to get Bing and NRC sentiment scores for each review for each airline. The final data frame included the original review, the airline, and the scores for all three sentiment lexicons. That final dataframe looks like this:
| airline | comment_number | final_afinn_value | bing_positive_count | bing_negative_count | nrc_anger_count | nrc_anticipation_count | nrc_disgust_count | nrc_fear_count | nrc_negative_count | nrc_positive_count | nrc_sadness_count | nrc_surprise_count | nrc_trust_count |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| American Airlines | 1 | -6 | 0 | 2 | 2 | 2 | 2 | 2 | 2 | 4 | 1 | 1 | 4 |
| American Airlines | 2 | -2 | 1 | 3 | 0 | 1 | 0 | 1 | 1 | 3 | 1 | 0 | 2 |
| American Airlines | 3 | -26 | 7 | 26 | 7 | 3 | 3 | 5 | 17 | 11 | 7 | 3 | 5 |
| American Airlines | 4 | 5 | 3 | 0 | 0 | 1 | 0 | 0 | 0 | 3 | 0 | 0 | 1 |
| American Airlines | 5 | -8 | 1 | 5 | 2 | 4 | 1 | 1 | 4 | 9 | 2 | 2 | 2 |
Visuals
To get a better understanding of which airlines are seeing less negative reviews, the following visuals were created.
Airlines by Average Afinn Score
For the top 10 North American Airlines, the following graph was created in order to show which airlines saw the most negative and the least negative reviews:
As a result, the assumption can be made that the most criticized North American airline is Spirit Airlines, which may not come as a surprise. As for the least criticized, Delta had the highest average Afinn Score.
Variation of Afinn Scores
To get a better understanding of Affin score variation, the following boxplot was created with each airline getting a unique boxplot with its review aggregate variation shown:
Based on these results, there are a few airlines that stand out on having more variation than others. Those airlines include, Jetblue Airways, American Airlines, and Allegiant Air. Something to note, in the last visual, Delta Air Lines had the highest average score, but it appears here that Jetblue Airways and United Airlines have higher median scores, meaning they had less variation.
Airlines by Average Bing Score
Next, taking a look at the Bing lexicon, which compares positive and negative words:
These results show which airlines receive the most positive and negative reviews. The counts below indicate the average number of both positive and negative words, per review, summarized by airline. While Frontier appears to the fewest number of negative words, it appears that they also receive the fewest amount of positive words as well. United Airlines averaged the most positive words of these 10 airlines according to Bing Sentiment.
Summary
In today’s competitive airline landscape, where customer satisfaction is paramount, understanding and addressing criticism is key to maintaining a positive brand image and staying ahead. Airlines often face intense scrutiny, with passenger reviews providing invaluable insights into areas for improvement. By using sentiment analysis techniques like AFINN, BING, and NRC lexicons, airlines can systematically analyze customer sentiment from reviews, identifying common pain points and gauging overall brand sentiment. This analysis not only helps airlines pinpoint areas for improvement but also allows them to benchmark against competitors, identifying which airlines receive the least criticism and excel in customer satisfaction. Armed with this knowledge, airlines can proactively address concerns, tailor services to meet customer expectations, and ultimately stand out in a crowded market.