For this assignment, we will be working on understanding the customer service experience offered by a hypothetical airline. When support is required, a customer initiates contact with the airline, either over the phone or using an online platform. Once connected, an agent of the airline works with the customer to understand the problem. When the interaction is complete, the agent creates records about the case. This includes the overall category of the issue along with whether the problem was resolved.
The airline would like to better understand the differences between support provided over the phone and online. With different types of interactions, the agents require different kinds of training, and the costs and resources are different in each modality. With this in mind, the airline is quite interested in comparing the quality of each type of service with regard to how well the cases can be resolved. They are also generally interested in improving their customer service and better understanding the experience.
In this assignment, we will be working with the information provided and consider the question of how to best understand the customer service experience.
For each call, the following characteristics were measured:
category: Each case was classified according to the type of service that was requested by the customer.
service: This indicates whether the support was provided by phone or online.
waiting.time.minutes: This records how long the customer waited to begin interacting with an agent.
session.minutes: This records how long the customer interacted with the agent.
customer.demeanor: This records the agent’s perception of the customer based on the tone of the conversation.
resolved: This measures whether the agent was able to solve the customer’s main concern in the interaction.
Based upon the information above and the data provided, please answer the following questions. When numeric answers are requested, a few sentences of explanation should also be provided. Please show your code for any calculations performed.
This section of the report is reserved for any work you plan to do ahead of answering the questions – such as loading or exploring the data.
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionlibrary(ggplot2)
library(tidyr)
library(readr)
airline_customer_service <- read_csv("airline customer service.csv")##
## ── Column specification ────────────────────────────────────────────────────────
## cols(
## category = col_character(),
## service = col_character(),
## waiting.time.minutes = col_double(),
## session.minutes = col_double(),
## customer.demeanor = col_character(),
## resolved = col_logical()
## )What is the primary research question that will help the airline? Make sure to state this in terms of a measurable quantity.
The airline is interested in comparing the quality of two modalities of service - over the phone or through an online platform - with regard to how well customer service cases can be resolved. So the primary research question that will help the airline is whether there is a difference in the resolution rate between over the phone cases and online cases.
What kind of research study can be conducted with the available data?
Based on the available data, an observational research study can be conducted. We are only able to analyze the outcomes of customer service calls that already happened, not trying to affect them by applying treatment.
Are there any drawbacks to the kind of study you plan to undertake?
One of the drawbacks of an observational study is that it can only identify association and not causation. We have no control over the subjects and therefore cannot assign them to a group. Another major limitation is the potential for unmeasured confounding variables; in this study, we will not be able to control for other variables that could influence the customer’s decision to initiate contact over the phone or online and the resolution success or failure of an inquiry. These variables could include things like customer demographics, customer history with the airline, and severity of the customer service issue.
If you could devise your own experiment, what would you do differently?
If I could devise a new experiment, I would create a randomized trial. For instance, I might have a pool of study participants that I randomly split into a control group and a treatment group. I would devise a set of customer service inquiries that is representative of what the airline’s customer service agents regularly receive. Then I would randomly assign inquiries to the participants in each group, controlling for category and for potential unmeasured confounding variables such as those described in Question 3. Ultimately I would have the participants test out their inquiries by calling in if they are part of the control group or using the online system if they are part of the treatment group, and measure the waiting and session time and whether the inquiry is successfully resolved. This design would make the experience consistent between the phone and online groups and ensure that any resulting differences between the groups are caused by the modality.
Perform a statistical test that would analyze the relationship between the independent and dependent variables. In this test, do not consider any other variables. What would you conclude from this test?
To analyze the relationship between the independent variable (phone or online) and dependent variable (inquiries resolved), it is appropriate to perform a two-sample test of proportions. “Resolved” is a logical variable, and we want to find the proportion of inquiries for which it is true in both groups, and determine if there is a significant difference between them. The result of the test estimated the proportion of resolved inquiries at approximately 77.0% online and 79.9% over the phone. The p-value was 3.299e-11, which is well below our significance level of .05, so we reject the null hypothesis that the resolution rates between the two groups are equal. These results mean that there is evidence of a significantly higher resolution rate over the phone than online.
service_totals <- airline_customer_service %>%
group_by(service) %>%
count()
service_totals## # A tibble: 2 x 2
## # Groups: service [2]
## service n
## <chr> <int>
## 1 Online 11372
## 2 Phone 38628resolved_counts <- airline_customer_service %>%
group_by(service, resolved) %>%
count()
resolved_counts_pivot <- resolved_counts %>%
pivot_wider(names_from = resolved, values_from = n)
resolved_counts_pivot## # A tibble: 2 x 3
## # Groups: service [2]
## service `FALSE` `TRUE`
## <chr> <int> <int>
## 1 Online 2613 8759
## 2 Phone 7765 30863successes <- c(8759, 30863)
totals <- c(11372, 38628)
prop.test(successes, totals)##
## 2-sample test for equality of proportions with continuity correction
##
## data: successes out of totals
## X-squared = 43.991, df = 1, p-value = 3.299e-11
## alternative hypothesis: two.sided
## 95 percent confidence interval:
## -0.03751558 -0.01999422
## sample estimates:
## prop 1 prop 2
## 0.7702251 0.7989800Apart from the independent and dependent variables, how should we think about the other variables in the data, and what would be the best way to consider them in the analysis?
We can think about the other variables in the data as measured differences. These may be potentially confounding and we want to adjust for them in analyzing the relationship between the independent and dependent variables. To do this, we will utilize a multivariable adjusted model. This model will account for measured differences and allow us to evaluate the effect of the independent variable.
What would be an appropriate way to incorporate these other measured variables into an analysis of the relationship between the independent and dependent variables?
Our dependent variable, RESOLVED, is binary. Our independent variable, SERVICE, is categorical. Our other measured variables are both categorical and continuous. We can use a logistic regression model to predict the resolution status based on the dependent variable and these other measured variables.
Create a model that estimates the effect of the independent variable on the dependent variable while incorporating the other measured variables. Show the estimates and any measures of significance.
This model shows that when we incorporate the other measured variables, all of these measured variables as well as the independent variable influence the dependent variable. Seen in the code output below are the p-values of each variable - all below our .05 significance level - the estimates which show how much each variable positively or negatively impacts the resolution likelihood, and the overall AIC of the model. The AIC describes how good of a fit the model is, with a lower AIC indicating a relatively better model when selecting between multiple options.
model <- glm(resolved ~ category + service + waiting.time.minutes + session.minutes + customer.demeanor, family = "binomial", data = airline_customer_service)
summary(model)##
## Call:
## glm(formula = resolved ~ category + service + waiting.time.minutes +
## session.minutes + customer.demeanor, family = "binomial",
## data = airline_customer_service)
##
## Deviance Residuals:
## Min 1Q Median 3Q Max
## -2.1142 0.5504 0.6339 0.7092 1.0491
##
## Coefficients:
## Estimate Std. Error z value Pr(>|z|)
## (Intercept) 0.506727 0.133709 3.790 0.000151 ***
## categoryService -0.269141 0.051846 -5.191 2.09e-07 ***
## categoryTechnology -0.660596 0.041337 -15.981 < 2e-16 ***
## categoryTickets -0.197089 0.038007 -5.186 2.15e-07 ***
## servicePhone 0.065854 0.030300 2.173 0.029750 *
## waiting.time.minutes -0.026128 0.002029 -12.877 < 2e-16 ***
## session.minutes 0.103959 0.013089 7.943 1.98e-15 ***
## customer.demeanorNeutral 0.590529 0.059191 9.977 < 2e-16 ***
## customer.demeanorNice 0.457216 0.048323 9.462 < 2e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## (Dispersion parameter for binomial family taken to be 1)
##
## Null deviance: 51071 on 49999 degrees of freedom
## Residual deviance: 50359 on 49991 degrees of freedom
## AIC: 50377
##
## Number of Fisher Scoring iterations: 4Are there any other concerns with regard to this study and its design?
One concern with the model in Question 8 is that it seems like our other measured variables may have more of an impact on the dependent variable than our independent variable does. This could mean that the association we saw between service type and resolution could actually be attributed to one or more other variables. As is the case with the independent variable, we also can’t establish causality between any of these measured variables and the dependent variable, because we don’t know how they are all interacting with each other.
Now imagine that the conclusion you drew from analyzing the data is actually incorrect. In fact, the real effect of the independent variable is exactly the opposite of what you demonstrated. How would you explain what happened?
It is possible that while the group of inquiries taken by phone had a higher rate of resolution than the group of inquiries taken online, the phone agent process itself is less effective at resolution than the online agent process. This could happen because of one or more other variables that differed between the two groups and caused the resolution rate for the phone group to surpass the online group, for example if customers were using the phone channel for easier to resolve inquiry types than on the online channel. That is why, in attempting to establish causation, it is important to design a study or experiment in which we can keep those other factors constant.
The customer service organization is also interested in understanding the quality of its work and the overall satisfaction of the customer. At the conclusion of the call, there is an opportunity to conduct a survey. What is the best way to implement this idea?
To implement a survey at the conclusion of the call, it may be a good idea to notify customers before the call that there will be an opportunity to answer some questions about their experience after the call and encourage them to stay on the line. This request should be specific with how much effort is required - for instance, please stay on the line for a one-minute survey or a quick 3-question survey. Then the survey should begin promptly after the customer service agent hangs up, so that the customer does not lose patience or forget to stay on for the survey. Providing the survey immediately after the call results in the greatest likelihood of response and the most authenticity. It may also be important to explain that the survey is confidential to reduce bias in the responses. We should ensure that the survey results are linked to the call so that we can get more details on that specific inquiry and the agent performance.
What are some possible topics that you might ask about? Select three potential areas and briefly discuss why these are important to gather information about.
Three areas that it may be useful to ask about to supplement the data already collected from the study are wait time, session time, and overall satisfaction. Wait time and session time are important to gather information about because while we have the minutes measured, we cannot gauge customer perceptions about what they deem to be acceptable service from the minutes alone. Gathering this information about how happy customers are with their wait time and session time will also help us understand any possible links between these measures and customer satisfaction. Overall satisfaction is also valuable information because it is a more meaningful measure of success than simply if the inquiry was resolved or not, and a specific survey question about satisfaction is more reliable in gauging customer satisfaction than the agent’s perception of customer demeanor.
For each of the three areas that you selected above, design a survey question. Keep in mind that the design should be appropriate for the setting. Provide the question, the possible answers, and the meaning of the answers.
1 - Strongly Disagree 2 - Disagree 3 - Neither Agree Nor Disagree 4 - Agree 5 - Strongly Agree
1 - Strongly Disagree 2 - Disagree 3 - Neither Agree Nor Disagree 4 - Agree 5 - Strongly Agree
1 - Strongly Disagree 2 - Disagree 3 - Neither Agree Nor Disagree 4 - Agree 5 - Strongly Agree
In this setting, how many questions would you ask the customer? Explain your answer.
In this setting, I would only ask the three questions. On a phone survey, it takes a while to read the questions and collect responses. We don’t want customers to have to wait too long that they become frustrated or abandon the survey. Three questions allows us to collect information about three pertinent areas to the customer service organization - wait time, session time, and overall satisfaction - in a timely enough fashion that will hopefully maximize the response rate.
The airline has a larger number of questions that it would like to ask. What would be your strategy for gathering all of this information? Explain your answer.
One strategy for gathering information on a larger number of questions while still keeping surveys short would be to have multiple different sets of survey questions that are randomly assigned to customers. So for example, if there were 9 questions the airline wanted to ask, we could ask a third of the respondents questions 1-3, a third of the respondents questions 4-6, and a third of respondents 7-9. Another option would be to create a longer survey incorporating all of the questions the airline wants to ask.
In this context, what are the advantages of a longer survey, and what are the benefits for a shorter survey? Explain your answers.
Longer surveys would provide insight into more areas of interest. They allow for more detail and a more comprehensive picture of each customer interaction, for those that answer the survey. Shorter surveys have higher response rates because they take less time and less effort from the customers. So shorter surveys will usually have a larger sample of responses, but can only measure a few questions. Shorter surveys also may have higher accuracy of responses than longer surveys, because after going through many questions people tend to stop paying attention and choose arbitrary responses either intentionally or accidentally.
If the airline’s managers are adamant about asking all of the preferred questions, what are some alternatives to this automated survey?
If we need to ask all of the preferred questions, we can change the format of the survey to be better suited. For example, we can send a survey via email, not necessarily immediately after the customer service interaction but within a few hours or days. This will allow for more questions as well as longer, potentially open-ended responses, which can provide invaluable information and recommendations for improvement. To do this, it is important to identify the customer on the call to allow for a follow up: if a customer has provided an email address or reservation number associated with their account, it will be very easy to find them after, but if a customer is inquiring about a technology issue, for instance, it may be necessary to identify them via phone number or other method.
Which customers would be more likely to participate in the automated survey after the customer service call, and which customers would be less likely?
In general, customers who have a very positive or very negative experience are more likely to participate in customer experience surveys than customers who have a relatively neutral experience. The customers who have a negative experience are incentivized to take surveys because they want to improve the experience, and they also might be frustrated and want to take out their frustration by providing negative responses. Customers who have a very positive experience with an agent might want to stay on for the survey as a show of gratitude to the agent. This may cause a participation bias, because the responses from these customers who are eager enough to take the survey might not be representative of the customers who are not eager enough to take the survey.
How reliable would you consider the information that comes from the automated survey to be?
The information that comes from the automated survey will be fresh in the customer’s mind because it immediately succeeded the customer service interaction, so it will likely accurately depict their perception of the interaction. However, customers might be experiencing heightened emotions immediately after the interaction, so if one aspect of the interaction went poorly and the customer is unhappy, they may negatively rate other unrelated aspects of the interaction as a result. It is also important to take into account that while we designed the automated survey to maximize responses, there will still be some non-response bias from customers who do not take it.
What else could you recommend to the managers of the airline’s customer service center to help them achieve the stated goal of understanding the quality of its work and the overall satisfaction of the customer? Provide a number of strategic recommendations that are actionable, measurable, and amenable to experimentation.
One thing we can do within the surveys is to test new ideas and see how customers respond to them. If the managers of the customer service center have some thoughts about a new product or service or technology to introduce, we can take advantage of the surveys already being sent out to ask questions like “is this something you would use if we had it?” Another thing along those lines would be to perform an experiment within the surveys - customers can be randomly split into treatment and control groups and given different questions, so then we can measure the estimated impact of one decision over another. Thirdly, the managers could try utilizing the agents for additional insight. Similarly to how the agents measure the perceived demeanor of the customer, they could provide feedback on how smoothly a customer interaction went or how well prepared they felt for the question they received. As long as it is anonymous and agents don’t feel that they will be penalized for describing things they need help with, then the managers can implement new training procedures or provide additional resources. These recommendations can all be tested out on a small sample of customers and/or agents and the results measured before implementing them into the general procedure if they seem to be successful in providing actionable insights.